OOCCMENT RESUME 



ED 069 276 



JC 720 261 



AUTHOR 

TITLE 



INSTITUTION 

SPONS AGENCY 

PUB DATE 
NOTE 



Brooks, Halter 

Nor Cal Research Group Vocational Education Study: A 
Field Study To Determine Characteristics of Most 
Successful Vocational Education Programs. 

Northern California Community Colleges Research 
Group. 

California Community Colleges, Sacramento. Office of 
the Chancellor. 

30 Sep 72 
130p. 



EORS PRICE 
DESCRIPTORS 



IDENTIFIERS 



MF-$0.65 HC-J6.58 

Communication (Thought Transfer} ; "‘Community 
Colleges; *Conferences ; Data Analysis; *Field 
Studies; Hypothesis Testing; Interviews; Program 
Evaluation; Projects; ^Questionnaires; Tables (Data); 
Techniques; * Vocational Education 
^Delphi Technique 



ABSTRACT 

A field study to describe common characteristics of 
three •'roost successful" vocational programs in each participating 
northern California community college is presented. The study was 
divided into distinct phases. In Phase I, a procedure for identifying 
three "roost successful" programs on each college campus was developed 
and implemented. In Phase II, characteristics of "successful" 
differences. The technique used in Phase I is the Delphi technique, 
which is a means of structuring communication to overcome these 
detrimental effects of open communication. Phase I began with a 
project planning conference whose primary objectives were to acquaint 
college representatives with the objectives of the study# to make 
necessary modifications in the plan, and to enlist local college 
support for the field study. In all cases, agreement between Delphi 
panels was sufficient to identify three programs which met the 
project definition of success. In Phase II of the study, a 
questionnaire was developed and administered in a standardized 
interview of first-line administrative personnel of successful and 
"other" vocational education programs ^ Hypotheses tested were: (1) 
Vocational programs identified by Delphi panels as roost successful 
have common identifiable characteristics; and (2) Vocational programs 
identified by Delphi panels as most successful differ from "other" 
programs in program characteristics. The essential results of the 
Phase II questionnaire are shown in tabular form. (CKl 
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PREFACE 



Since ve who participate in the northern California Conmunity 
College Research Group believe its activities are extremely Important 
to our colleges, mention should be made of the nature of the organize** 
tlon at the outset of this report. The **Wor Cal*' Research Group is an 
Informal and voluntary association of community college educators who 
carry primary responsibility for, or maintain a strong interest in, the 
institutional research and development function of their college. The 
basic purposes of Kor Cal are to encourage cooperative research among 
colleges and to exchange information on research projects and innovative 
programs . 

Althou^i Nor Cal originated the idea for this project, it cannot 
claim it as its own. Support from the vocational education deans of our 
colleges, the California Community College Chancellor’s Office, and the 
State Department of Education was essential to project development. The 
idea for this cooperative project came from one of the bi-monthly meet- 
ings of the Nor Cal Research Group at Cabrillo College in the spring of 
1971> Lorlne Aughlnbaugh, Assistant Dean of Research from American River 
College, and I accepted the responsibility for developing a specific re- 
search plan and applying to the California Community College Chancellor’s 
Office for VEA funding. At the next meeting of the organization at Solano 
Community College, the plan was presented to Nor Cal members and was adopt- 
ed as a cooperative research project for the 1971-72 school year. 

Upon application to the Chancellor’s Office for funding, ve were ex- 
tremely fortunate in having Dr. Bill Morris, Consultant on Evaluation of 
Vocational. Education for the Chancellor's Office, and Dr. Earnest Neasham, 
Educational Research Evaluation Consxxltant for the State Department of Edu- 
cation, review oxir proposal. Both saw merit in the proposal and helped us 
to clarliy and sharpen our research design. Once the project was funded 
both agreed to sit on our Project Advisory Committee where they contributed 
greatly as individuals to the development of the project. 

It was obvious to us at the outset that the project could never suc- 
ceed without close cooperation of the vocational education deans of our 
member colleges. I4r. Iloyd Livingston of Shasta College, Dr. Louis Quint 
of American River College, and Dr. Harry Shortess from the Peralta Communi- 
ty College District Office consented to Join our Project Advisory Committee. 

O 

ERIC 



6 ' 



These men were extremely helpful in enlisting the support of the voca- 
tional education deans of the participating colleges, but we depended on . 
them more for their insight into vocational education programs and pro- 
gram management. All seven members of the Nor Cal Research Group Steer- 
ing Committee served on the advisory committee. They were: Mrs. Lorine 

, t 

Aughlnbaugh, Assistant Dean of Research, American River College: Dr. 

Dayton Axtell, Counselor and Psychometrist , Merritt College; Mr. Walteiy 
Brooks, Director of Research, Shasta College; Dr. Patricia Hertert, In- 
structional Resources Consultant, Yosemite Junior College District; Mrs. 
Virginia Murdoff, Dean of Students, Napa College; Dr. Pavil Preising, 

Director of Institutional Research, San Jose City College; and Dr. Lance 
Rogers, Director of Center for Independent Learning, City Collesge of San 
Francisco. 

Napa College acted as the fisced agent for the project and both Vir- 
ginia Murdoff, Nor Cal Treasurer, 6ind Andy Peterson, Napa College Busiiuess 
Manager, suffered graciously with the expense vouchers and requests for 
payment emanating from colleges 'edl over the north state. 

During the data collection phase of this project, Mr. Blaine Wlshart 
served as project director. The extremely competent and thorough anedysls 
of the data, which is reported in this study, is the work of Dr. Fred Da- 
gcnais of the Far West Laboratory for Educational Research and Development 
in Berkeley. 

One of the most farsighted decisions of the advisory committee was to 
select Dr. Ben Gold, Director of Institutional Research at Los Angeles City 
College, as project auditor. Although Dr. Gold had never conducted an au- 
dit of an educational research project prior to this one, his approach to 
the task ouf,ht to serve as a model for those with extensive experience in 
the role who see themselves as project historieuis. Dr. Gold attended all 
key meetings and conferences for a flrsthaind look at what was occurring. 

He not only summarized and evaluated progress, his Interim audit reports 
offered us specific, positive s\iggestions. His guidance often gave us our 
bearings and helped us to avoid the more monumented blunders we would have 
made without him. 

Ultimately however, the project was not an individual accomplishment; 
it was undertaken as an extra Job for almost everyone involved. Ho college 
representative was required to participate and few of those who did received 
more than their travel expenses for doing so. We began the project with the 



hope that what we were doing was an important first step in developing 
management information for vocational educators * and with the certain 
knowledge that we must learn to learn from one another in community col- 
leges if we are to be effective in our educational role. 
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A FIELD STUDY TO DETERMINE CHARACTERISTICS 
OF MOST SUCCESSFUL VOCATIONAL EDUCATION PROGRAMS 

Historically, California community colleges have controlled the de- 
velopment of their own vocational or occupational programs. A few pro- 
grams must meet state standards for licensing but in most cases, colleges 
have been relatively free to initiate those programs which they choose to 
initiate. Curriculum, facilities, and instructional, staff have been pro- 
vided according to local standards and budgets. To say that community col- 
lege administrators and boards of trustees prefer such an arrangement is 
probably a gross understatement; in fact, state-wide standards or policies 
are rarely adopted by community colleges without a protracted struggle. 

California community colleges have guarded their local autonomy in 
the decision making process for a pui*pose. Externally imposed standards 
or guidelines are seen as a serious threat to the basic educational role 
of the community college. Most comnunity college educators believe that 
the phenomenal success of the institution is based upon its ability to tai- 
lor policies and programs to fit local needs. This attitude of emphasiz- 
ing community service while rejecting outside controls has undoubtedly con- 
tributed greatly to the emergence of the community college as a new form of 
higher education; but like all virtues, local control may have its negative 
sides. One of the negative aspects of this en^hasls on local autonomy is 
that it may tend to make \is more provincial in our outlook and less effec- 
tive in program development than we might be. 

Community colleges generaJ.ly do a good Job of meeting the needs with- 
in a district, but if published results of Institutional research studies 
are a vctLld indicator, they are slow to share their own experience or pro- 
fit by the experience of other colleges. An inspection of the information 
available through the ERIC system on all aspects of community college edu- 
cation will verify our lack of formal communication. Relatively few studies 
of significant aspects of educational programs or practices are initiated 
by the colleges themselves. This is especially true of studies which cross 
district boundaries. Most cooperative studies are undertaken by outside 
agencies or graduate students in search of a degree. This condition is as 
prevalent in vocational education as it is in other areas of community col- 
lege education. 



The iiunber of vocational programs offered in California community 
colleges has Increased two and one half times in the past seven years 
(^♦9 75IO-UI), but very few realistic guides are available to the vocation- 
al administrator in developing new programs or modifying old ones. New 
programs are usually the private inspiration of a local vocational educa- 
tor combined with ideas borrowed from similar programs offered by the near- 
est neighboring college; once begun, programs are rarely modified. This 
field study, cooperatively undertaken by northern California community col- 
lege vocational administrators and research officers, is a first attempt to 
communicate more systematically about vocational programs and thereby im- 
prove existing management information. 

PURPOSE OF THE STUDY 

The specific Intent of this field study was to describe common char- 
acteristics of t’crec "most successful" vocational education programs in 
each participatiUtt h^a'thern California community college. It was felt that 
a careful analysis of the chai'acterlstics of each successful program would 
lead to the discovery of characteristics common to all successful programs 
and ultimately, meaningful management information for vocational adminis- 
trators might be developed. When comparisons were made, successful programs 
were ccxnpared with "other" programs or programs in general which were not 
selected as successful. It is Important to note that in no case were suc- 
cessful programs compared with unsuccessful . It is far more difficult to 
differentiate successful programs from programs in general than it is suc- 
cessful programs from unsuccessful. Differences which emerge in this kind 
of analysis are far more difficult to identify but are possibly more impor- 
tant than they would be if successful and vinsuccessful were compared. The 
decision to concentrate upon characteristics of successful programs was 
partly a matter of expedieiicy. With one year to conplete the study, limita- 
tions had to be imposed upon the scope of the study. It was assumed that if 
a complete analysis of program characteristics was not possible that an em- 
phasis on the analysis of successful programs would bear the greatest fruit. 

The diplomatic implication of limiting the project to identification 
and analysis of successful programs was not ignored. To begin the project, 
it was necessary to solicit the cooperation of representatives from each of 
twenty independent California community colleges who would be called upon 
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to supply staff, time, and resources voluntarily. Obviously, in this sort 
of undertaking a positive approach to program analysis is unquestionably 
appropriate. The primary danger in such an approach, however, is that the 
reader will conclude that simply because a given characteristic did not dis- 
tinguish successful programs, it was not necessary to success. Character- 
istics which were common to both successful and other programs night not 
stand out in this study; that is, those characteristics which were a neces- 
sary element of all programs wotild not be identified. 

FLAW OF THE STUDY 

This field study was divided into two distinct phases. In Phase I of 
the study, a procedure for identifying three "most successful" programs on 
each college campus was developed and implemented. In Phase II of the study, 
characteristics of "successful" and "other" vocational programs were de- 
scribed and analyzed for significant differences. An overview of the plan 
of the study is presented dlagranunatically in Appendix 1. 

PHASE I 

Planning Conference 

Phase I of the study began at Shasta College on October 29, 1971 with 
a project planning conference. The specific objectives and activities of 
the planning conference are shown in Appendix 2. The primaty objectives 
of the conference were to acquaint college representatives with the objec- 
tives of the study, to make necessary modifications in the plan, and to en- 
list local college support for the field study. 

Vocational Program Definition 

In this study, a vocational program was defined as one which was listed 
in the college catalog as a vocatloneil major and was designed primarily to 
give the student employable skills in a specified occupational area in two 
years or less. No stipulation was made with regard to standards to be met 
or degrees to be achieved. It was recognized that many college catalogs con** 
tained descriptions of vocational programs which were out of date or were not 
actually offered as a cohesive program. It was left to the discretion of the 
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vocational education dean from each participating college to submit a 
list of programs offered on his college canqpus which met the project 
definition. 

Identifying Successful Vocational Programs 

It was decided that each college would identify three most success- 
ful vocational education programs throu^ a local panel of Judges using a 
"Delphi" communication structure. In situations where no objective cri- 
teria are available, it has been found that opinions of individuals can be 
pooled to make better Judgments tlian the same people would make individu- 
ally or through traditional means of group discussion. Recent experiments 
by researchers at Rand Corporation in Santa Monica have combined certain 
principles of communication which are useful in situations where objective 
data is not available and Judgments must be made on the basis of opinion. 
Collectively, these principles have come to be known as the Delphi tech- 
nique. Even with the exotic name, however, Delphi depends upon well>-known 
psychological principles rather than metaphysics for its accuracy. In es- 
sence, it is a way of searching out the opinion in a group which* most close- 
ly approximates objective measures. 

Qnployment of the Delphi technique in most cases requires no great 
technical skill. Althou^ the specifics change from situation to situation, 
necessary Instruments and procedures are usually uncomplicated. Partici- 
pants are asked to state their opinion on a proposed question in writing. 

The t'>p!nlon is given anonymously without consxiltatlon with other panelists . 
All statements are collected, duplicated, and a conqalete set of statements 
is returned to each member of the group for consideration. Participants 
are asked to indicate agreement or disagreement ^/ith the statements in some 
fashion. These responses are again collected, reproduced, and distributed 
to the total group. The process continues for a predetermined nxunber of 
rounds or Iterations, or until group consensus is reached upon the state- 
ments . 

While on the surface the method is a seemingly circuitous means of 
arriving at a group Judgment, it is Intended to overcome some formidable 
obstacles usually present in open communication among group members . In 
open communication, especially face-to-face communication, several factors 
combine to distort opinion and reduce accuracy: statements of hi^ status 

group members are likely to be viewed by other members as more Important 
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or accurate than they reeJ.ly are. Dominant members of the group, those 
that like to express opinions and talk, are likely to affect group Judg- 
ment out of proportion to the quality or accuracy of the contribution. 

In open communication, the individual quite often clings to and defends 
a public statement even agsdnst strong evidence that the initial opinion 
was in error. Once majority opinion is known in open communication, how- 
ever, there is great pressure to accept the majority Judgment regardless 
of accuracy. Finally, open communication usually involves a face-to-face 
encounter and requires an assembly of people in the same place at the same 
time . 

The Delphi technique is a means of structxiring communication to over- 
come these detrimental effects of open communication. It is a controlled 
process of interaction to make the most of group Judgments. Delphi assumes 
that if initi£d opinions are expressed anonymously, there is less tendency 
to be affected by dominant or high status members of the group. Each mem- 
ber of the group is allowed to see how eveiy other member recu;ts to state- 
ments without pressure from the majority, and with no public commitment 
of his own. It is assumed that under these circumstances the participant 
is in the best possible position to moke a new Judgment solely on the basis 
of the communication. Finally, but by no meeins least inq>ortont in a prac- 
tical sense, it is not necessary to have group members meet face to face in 
a Delphi exercise. 

VIhile questionnaires and response modes vazy, certain key elements are 
common to all Delphi procedzires. They are: 

1. Anonymity of respondents 

2. Iteration and contz'olled feedback of individual Judgments 
to the entire panel 

3. Grouping responses in describing panel consensus 
Delphi Petnels 

At the fall planning conference, it was agreed that two panels would 
be selected on each college campus and would work independently at the 
task of identifying most successful progzams. This was an innovation in 
Delphi methodology introduced dzzring the project to include a reliability 
check on the work of the panels. The use of two independent panels was 
also critical to the project opezational definition of program success. 



Both the reliability check and the definition will be described in later 
sections . 

An attempt was made to keep the panels con^arable in their structure 
by including on both panels representatives from each of the following 
categories : 

1. Dean of Instruction, Associate Dean of Instruction, or 
Dean of Guidance and Admissions 

2. Member of College Board of Trustees or Member of Vocational 
Advisory Committee 

3> Vocational Counselor 

4 . Academic Counselor 

5. Instructor in Transfer Curriculum 

6. Instructor in Vocational Curriculum 

7 • Classified Admissions Office Personnel 

8. Two Students Enrolled in Vocational Programs 

9> Two Students Enrolled in Transfer Programs 
VJhere the organizational structure of the college did not agree with the 
categories indicated, substitutions were made which approximated the ide- 
alized panel . Where a participant in the category was \mavailable , the 
same rule applied. Most colleges taking part in the process were able 
to meet the project requirements quite closely. 

Delphi Instrument 

The Delphi instrument used in Phase I of this project to identify 
most successful vocational programs is shown in Appendix 3. Initially, 
the two separate panels selected on each campus were asked to Identify up 
to five vocational education programs offered by their colleges which they 
considered to be most successful, and to list reasons for their choice. A 
list of the vocational offerings which met the project definition was at- 
tached to the first round questionnaire. When first round questionnaires 
were returned, the name of each program Indicated by panel participants 
was transfered to a separate sheet of paper. The percentage of partici- 
pants choosing the program was shown at the top of the paper and all the 
reasons given for selecting the program were listed below it. 

In the second round of the Delphi process, all sheets were clipped 
together. Instructions on the cover sheet asked the panel participants 



to read what all other panelists had to say aoout programs selected and 
make a ne\/ Judgment. This time panelists were asked to select three most 
successful programs Instead of five. At the end of the second rounds the 
results of the two panels, which were working independently to identify 
most successful programs, were compared. S uccessful programs were defined 
as the three commonly identified by both panels with the greatest percent- 
age of a/yeonent within the panel. If agreement between the two panels was 
low at this point, the second round was repeated showing new percentages of 
group response discarding from the list of programs those not selected on 
the second round. In most cases, however, agreement between the two panels 
even on the second round was very lii^. The second round questionnaire as 
it was returned by one of the colleges is shoxm in Appendix 3. 

On the third round questionnaire, panelists were asked to rate the 
comments they and other panelists had made about programs. These ratings 
were made in terms of their relative inqportance to program success. The 
Round 3 questionnaire is shown in Apjendix 3. Results from the Round 3 
questionnaire were an important element in developing standardized inter- 
views in Phase II of the project . 

Panel Selection and Admi nistration of the Questionnaire 

Enlisting the cooperation of a broad cross section of people on each 
college campus who would be taking part in the Delphi panels was a most 
critical step in the first phase of the project. This was only possible 
with the cooperation of the vocational education dean and the research 
officer on each canpus. It is probably important to point out here that 
the sole reason why cooperation was possible is that the field study grew 
out of the thinking and the concerns of those who carried it out. Most 
of those who cooperated had contributed omething to the development of 
the study plan so that while they cooperated with other colleges , they 
were dealing with a locally originated study as well. In the first phase 
of the study, complete returns were obtained from l6 of the 21 participa- 
ting colleges. In those cases where results were not complete, data col- 
lection problems encountered were beyond the control of the college repre- 
sentative. 



Delphi Procedure Relia'bility 



The division of the cauipus participants into two panels working inde- 
pendently to identify most successfiil programs made a most useful relia- 
bility check possible. The reader may recall that the vocational dean from 
each participating campus was asked to furnish a list of all vocational pro- 
grams meeting the project definition. This list was attached to the Roiuid 1 
questionnaire in the Delphi process. It constituted a "universe" of choices 
for panel participants. For purposes of our reliability estimate, each of 
the two Delphi panels might be thoiight of as drawing an independent san^le 
of choices from a known population of choices. If we know the universe of 
choices that are possible (number of vocational programs attached to the 
Roxmd I questionnaire) and we also know the name and number of programs iden- 
tified by each of the panels at the end of the second round in the Delphi pro- 
cess, the probability of two independent panels selecting the same vocational 
program from the universe of choices can be mathematically calculated. The 
reliability test might be best described with a hypothetical exanple. 

Let us say that one of the two Delphi panels. Panel A, after the second 
round of the Delphi process has reduced the number of programs it identifies 
as most successful to 11. Assume further that the second panel. Panel B, work- 
ing independently of the first panel, has reduced its choices of most success- 
ful programs to 9. Finally, let us assume that the two panels have commonly 
Identified 6 vocational programs. This situation is shown diagrammatinally 
below: 




Ve can now calc\ilate directly the probability of these two panels making 
a common choice of 6 progreuns from a universe of 30 possible choices with 
the mathematical formula shown below where: n = number of programs in the 

universe of choices; n^ = number of programs finally selected by Panel A 
after two Delphi rounds; n^ = number of programs selected by Panel B after 
two Delphi rounds; n^ = number of programs the two panels have independ- 
ently selected in common from the universe of choices. 



/n \ (n^^- n^) ‘ (n - n^) 

■ ■ 7- r ■ 



/n \ _ Number of different combinations of common items that can 
\ n^/ ” be drawn from universe 

( n - n.\ _ number of ways to draw other than common items for 

^ = total number of ways to draw (n^) 

/n - n^\ _ number ways to draw other than duplicates (from 
V^Ug- n^) for (n^) 

hence n - n^ = mnaber elements left for (ng) 

ng- = number to draw from the (n - n^) 

^ number ways in general to draw (ng) 

Substituting numbers from o\ir hypothetical example into the fomrula: 



P = 



11 I 9 I [21.20»19-i8.17] 

6 I 5 I 3 I 130.29*28. 27*26.b«24.23*22. 21. 20] 



= .03 
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The formula indicates that the probability of the two panels selec- 
ting the six common vocational programs by chance alone is 3 in 100 or 
expressed differently P ^ .03. 

This hypothetical example shows close agreement between the two pa- 
nels in their choice of successful, vocational programs. If such proba- 
bilities were obtained with real data, we could have a good deeil of confi- 
dence that our instruments were understood and responded to in a similar 
fashion by most panel members; that our participants were matched fairly 
carefully in the two panels; and most importantly, that there is essential 
agreement among a broad cross section of the college community on those 
programs which deserve the designation "most successful". This reliability 
determinant has been computed and reported for the Phase I returns of each 
participating college. 

PHASE I - Results 

Table 1 on page 15 describes the basic results of the Phase I Delphi 
process. Six colleges sutanitted complete Phase I returns. In all cases, 
agreement between Delphi panels was svifficient to IdentilV three programs 
which met the project definition of success. Thirteen of the sixteen col- 
leges showed agreement between panels at P = .05 or less. One of the three 
colleges which did not reach a probability level of .05 iq>on initial admin- 
istration of the second round q.uestionnaire , readministered the second round 
questionnaire and reduced the probability level that a non-chance agreement 
was occurring from .lU5 to .0002. Ranking of the first three vocational pro- 
grams meeting the project definition for successful programs did not change 
for this college. It was assumed from this test that the procedure was suf- 
ficiently reliable to accept the Round 2 results from all colleges as a valid 
Indicator. 

In Table 1, colleges are identified by a number only. It was agreed at 
the outset of the project that programs Identified as most successful on in- 
dividual college campuses would not be publicly named. Each vocatloneJ. edu- 
cation dean participating in the study was furnished a list of the names of 
the three programs on each college campus. It was assianed that such a list 
might be valuable to the dean for visitation purposes when instituting or 
revising a simlleir vocational program on his own campus. 
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Although Table 1 is somewhat lengthy, it was felt that inclusion 
of the complete results of the first two Delphi rounds was warranted. 

Since the study deals with an experimental procedure which may have sig- 
nificance beyond the specific application in this study, a full report on 
the process is in order. Percentages in the table refer to the percentage 
of people in the panel selecting a given vocational program as most suc- 
cessful in each of the first two rounds of the Delphi process. A third 
column shows combined percentages only for those programs identified by 
both groups. 

A third round was included in this Delphi process but is not shown in 
Table 1. In the third round, panel members were asked to select statements 
which reflect most inroortant reasons for calling a progreun successful. While 
this information was important in the development of the Phase II standard- 
ized interview, it was not considered as meemingful as the data which have 
been presented. It was therefore not included in the context of this re- 
port but has been organized in Appendix 4. 
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Table 2 below abstracts those programs Identified as most success- 
ful in colleges and groups them according to vocational area. 



TABLE 2 FREQUENCY OF VOCATIONAL PROGRAM SELECTION 



Vocational Area Programs Identified 

Agriculture and 
Natural Resources 

Agriculture 

Horticulture 

Business and 
Business Management 

Food and Food Services 
Hotel and Restaurant Management 
Management Training 
Real Estate 

Clerical and 
Secretarial 

Office Occupations 
Secretarial Skills 
Secreteurlal Training 

Computer and 
Information Technology- 

Data Processing 

Criminal Justice 

Administration of Justice 
Lav Enforcement 
Police Science 

Health Services 

Associate Degree Nursing 
Dental Assisting 
Dental Hygiene 
Inhalation Therapy- 
Licensed Vocational Nursing 
Psychiatric Technician 
X-ray Technician 

Trade and Indus- 
trial Technology 

Aero Technician 
Auto Body Mechanics 
Auto Mechanics 
Heavy Duty Mechanics 
. Marine Diving Technician . 

Sheet Metal 
Welding 



Frequency 



1 

1 



2 

2 

1 

1 



1 

1 

1 



1 



1 

2 

It 



8 

1 

2 

1 

7 

1 

1 
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1 

1 

1 

1 

1 
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PHASE II 



In Phase II of the study, a questionnaire was developed and ad- 
ministered in a standardized interview of first-line administrative 
personnel of successful and "other" vocational education programs. 

Data from the questionnaires were coded' and analyzed . The two hy- 
potheses forming the basis of the study were tested. 

Hypothesis I - Vocational programs identified by Delphi 
panels as most successful have common identifiable char- 
acteristics . 

Hypothesis II - Vocational programs identified by Delphi 
panels as most successful differ from "other" programs in 
program characteristics . 

Basis of the Standardized Interview 

The questionnaire used in the steuidardlzed interview conducted in 
Phase II of the study was developed from two major sources. First, at 
the initial planning conference, vocational education deans were asked 
to contribute cleeurly worded statements on 3 x 5 cards indicating their 
opinion on what made vocational education programs effective. Each dean 
was then asked to rate all statements as to Importance. Conference par- 
ticipants then categorized the statements and attempted to estimate how 
each category ml^t be measured. Second, statements made by panel par- 
ticipants in the Delphi exercise on each college campus were collected. 
Duplicate or very similar statements were ccanbined. Statements were 
listed on 3 X ^ cards. A group of five Judges knowledgeable in vocation- 
al education programs and curriculum development then categorized these 
statements. Those statements Judged to be effects or outcomes of success- 
ful vocational education programs were separated, from statements Judged to 
be causal elements. As an example, the statement "This program is very 
popular with students on campus" would be Judged to be an effect or out- 
come of a successful program whereas the statement "Instructors in this 
program are very good" would be Judged to be a causal element. 

The cause and effect criteria was applied to the list of statements 
provided by vocational education deans. Onlj' those statements which were 
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Judged to be causal were considered as possible elements In the ques- 
tionnaire. The Phase II questionnaire was developed from those ele- 
ments of the vocational deans statements and the Delphi panelists 
statements which could reasonably be expected to be collected through 
the Interview process. 

Limiting data collection activities In Phase II of the project to 
a standardized Interview \ras a necessazy but less theui Ideal choice. 

Many suggested variables for Identifying successful vocational progrcuns 
could not be measured throu^ the Interview process. Under Ideal cir- 
cumstances with more time to develop the study, more direct measures of 
variables affecting vocational programs would have been made. A complete 
standardized Interview Is shown In Appendix 5. 

Definition, of "Other” Vocational Progr^ws 

"Other” vocational programs on the college campuses were Identified 
any vocationaJ. program meeting the project definition for vocational 
programs, but not selected .lolntly by panel partlclnants In the 'second 
round of the Delphi process . Researchers on each college caiqpus were 
asked to choose any three programs which met the definition for "other* 
programs, but no rigorous randomization process was recommended to the 
local colleges. Vhlle on most campuses a random selection process was 
carried out in the identification of "other" programs, no standard pro- 
cedure was introduced. There were practical reasons why this could not 
be accomplished, but the lack of a standardized procedure was considered 
to be one of the major design weaknesses of the study. 

Administration of the Phase II Questionnaire 

Six Phase II interviews were carried out on each participating col- 
lege campus. The first-line supervisor of the three vocational programs 
identified as most successful and the three programs identified as "other" 
programs were interviewed by the college research team. All interviews 
were then collected for data analysis. 
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PHASE II - Plan of Analysis 



The Phase II questionnaire yielded approximately 00 bits of infor- 
mation with which programs identified as successful could be contrasted 
and tested a^inst programs which were not identified as successful. 

In addition, 50 to 100 bits of information could be created through the 
use of "dummy" items or variables . 

The strategy which guided the analysis of these data was to select 
an e3q>edient design which could be comm'jnicated clearly to the intended 
audience. For this reason, chi-square and correlational statistics pre- 
dominate as analytical tools. 

Questionnaire Returns 

Seventy-two usable Phase II questionnaires developed from interviews 
were returned in time for the analyses which follow. These questionnaires 
described three vocational programs from each of twelve community colleges 
in northern California identified in round one as "successful'^ and three 
vocational programs from each of the same colleges which were not so 
identified. 

Condition of Data 

The vast majority of the questionnaires were completely filled out. 
Missing data were rare and generally fell into two categories where; 

1. the Information requested was not applicable to the program 

2. the information was not known to the respondent 

If criticism is to be made of the questionnaire, for design pxirposes,. 
it is that some of the items were not worded tersely enou^ and that some 
items were not easily coded in a form that is readily machine processable 
for tabulation purposes. Yet this coding "oversight" produced several 
benefits. Qw.estions that are designed primarily for electronic data pro- 
cessing are often mechanical and present an image which may appear unduly 
"cold" to the person responding to them. In addition, it is often diffi- 
cult to anticipate all responses with a package of prepared answers. In 
an e3q>loratory study such as this, the use of open-ended questions can 
often elicit Iniportsuit information from Informed and considerate respon- 
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dents. This point is illustrated by question 1 in the Program Manage- 
ment section. In this question, a large n?miber of combinations were 
recorded which could not have been emticipated. Perhaps the main canis- 
sion in the questionnaire was its failure to elicit ‘hard" data such as 
is gathered by nanagement information systems. Yet these data are not 
easily accumulated even by management specleillsts . Specific measures 
of program functioning would require more than one yeeu: to agree upon 
c«. -ccn definitions and collection methods. 

Coding of Data 

Within the limits of the questionnaire data was coded and analyzed 
in an appropriate manner. The codes used for each question are listed 
in Appendix 4. Briefly, all questions which could be answered with a 
continuous numeric variable were utilized eis recorded. The following 
exceptions were made: 

1. Successful programs were coded "1” and programs not so 
identified were coded ’'O'*. 

2. Page 1, question 5. The ratio of males to females was 
converted to a percentage of males. 

3. Page 1, question 6. Many questionnaires had both in- 
side etnd outside checked. This weis coded: within =1, 

both - 2, and outside = 3. 

U. Peige 1, questions 1 and 2 on Instructors. The number 
checked was used except for "more than 5" where a "6" 
was used. Where an explicit number was given (such as 
"8") the number given was used. Part-time instructors 
were coded with one part-time instructor equaling one 
half full-time Instructor. 

5 * Wherever a yes /ho answer was given, the no was coded 
"0" and the yes was coded "l". 

6. For those questions which were emswered with a check, 
a “l" was used for a check and a "0" was used for a 
blank. 

7 . Severed questions elicited comments or a list of items 
as in question 10 on page 6. If em item was mentioned 
or a comment was made, then it was coded "1" and a 
blank was coded "0". 

The use of these codes, both continuous and dichotomous, made pos- 
sible the correlational statistics eind the tests of hypotheses which are 
presented in Table 3. 
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Means and Standard Deviations 



Means and standard deviations are presented in Table 3* Means and 
standard deviations were computed separately for the 30 programs iden- 
tified as successful and for the 36 programs not so identified. Standard 
deviations have been tabled for continuous variables, but not for dichot- 
omous items or variables because these would only serve to clutter the 
table. Standard deviations can be computed from the means for any dichot- 
omous variable by application of the formula S.D. = / Npq where N = 36, 
p = the tabled mean, and q = (l - P). 

Correlations with ^Successful* ' 

(The reader who is familiar with correlational statistics may skip the 
next paragraph.) 

Complete correlational data is presented in Table 3. All variables 
to be correlated with "success" were coded in such a way that an increase 
in the number coded means "more of something". Since programs identified 
as successful are coded "l" and programs not Identified as successful are 
coded ’‘0", positive correlations mean that more or whatever is being cor- 
related with success tends to correlate with success . This can be illus- 
trated by Inspecting several itons in Table 3. It will be seen that the 
number of units required in the major and number of years the program has 
been offered on campus both correlate positively with success. That is, 
the programs identified as successful tend to require more units and tend 
to have been offered over a longer period of time. Item 3, having to do 
with state licensing of a program, is dichotomous. It tells us that 
state licensing procedures tend to correlate with success. That is, more 
often than not, success" is associated with state licensing requirements. 
Other items have correlation coefficients which are negative. For ex- 
ample, the coefficient for question 7 in the Placement and Recruiting 
section has a negative sign before it and illustrates a negative correla- 
tion where we can associate fewer provisions for remedial work with pro- 
grams identified as successful. Conversely, we can associate more pro- 
vision for remedial work with programs not identified as successful. 
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PHASE II - Results 



The essential results of the Phase II questionnaire are shown in 
Table 3 on the following several pages. The table is in general self- 
explanatory and relates specifically to items on the questicxinaire , al- 
though some items are paraphrased. The meaning of the capital letters 
following some of the correlation coefficients may not be immediately 
apparent to the reader. The capital letter is a code for correlation 
coefficient probabilities and is explained at the conclusion of the ta- 
ble on page 52. 

FVom Table 3 it will be seen that 20 of the 58 correlation coeffi- 
cients calculated for these data have a likelihood or probability of 
happening through the workings of chance of less than one in ten. Al- 
though this minimal "level of significance" or P < .10 is acceptable to 
many social scientists, the preferred "level of significance" is one 

ft 

chance in twenty or P < .05. The twenty items which are represented by 
these correlations, are the items on which we will concentrate in the 
analysis that follows the presentation of Table 3. 
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For continuous variables the t-test was used. For dichotomous variables the chi-square test was used 
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How many times in the past year have advisory 10.9^ 5.83 l6.79 5.83 -I5l4 1.30 

committee members assisted in student placement? 

(Codes for correlation coefficient probabilities: A = p < -001; B - p < -01; C = p < .05; D = p < .10) 



Table of Correlations 



The twenty items with acceptable correlations with success are listed 
in Table U. They have been grouped under the headings of Student, Course, 
Instructor, Advisory Committee, and Miscellaneous. The logic of these 
clusters of items will be evident in the exposition which follows. 

TABLE U ITEMS CORRELATED WITH SUCCESSFUL PROGRAMS 



General Correlationt 

Classification Item Description* Coefficient 



Student 


Nvimber of students in program 


.335 B 


Items 


Percent male 


-.3><6 B 




Interview prior to admission 


.280 C 


Student 


Wears a uniform 


.1437 A 


Visibility 


Takes most courses in same area 


.210 D 


Items 


Eats lunch together 


.196 D 




Takes classes £md work together 


.210 D 




Belongs to occupational clubs 


.250 C 


Course 


State licensing procedure 


.l4l47 A 


Items 


Off-campus facilities used regularly for class 


.2214 D 




Provision made for remedial work 


-.201 D 


Instructor 


Number of Instructors in program 


.355 B 


Items 


Number of instructors with recent experience 
(2 years) 


.327 B 




Nianber of instructors with recent summer ex- 
perience 


.196 D 


Instructor 


Classroom visitations made regularly 


.209 D 


Supervision 


Tenured teachers visited 


.213 D 


Items 


Non-tenured teachers visited 


.259 C 


Advisory 


Advisory committee met in 1971-72 


.2214 C 


Committee 


Number of advisory committee members 


.273 C 


Items 


Advisory committee members employed in the 
field 


.237 C 


• These titles 


are paraphrases . See thti> questionnaire in Appendix 5 for 



exact wording of the questions, t The letters after the correlation coef- 
ficients indicate levels of statistical significance with probabilities of 
A<.001, B<.01, C<.05, and D<.10. 



Student Items 

Table k shows sex as one item which correlates significantly but nega- 
tively with success in this category. Although some of the strength of this 
correlation coefficient can be attributed to sampling problems (e.g., more 
"female" programs were selected as successful) it is likely that the sign 
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of the coefficient is correct. The negative correlation of "inale” ratio 
would have been positive if we had said "female” ratio. From a feminist 
point of view, the wording of this question may have been but another 
manifestation of sex discrimination, yet the negative sign served to draw 
our attention to this variable first . Since the correlation is rather 
stroig, as our correlations go, we must seriously consider the possibil- 
ity that the programs local panels selected as successful tend to have 
more female students. The implications of this are profound and cannot 
be adequately discussed here. One implication suggests consideration of 
other items such as occur under the Student Visibility heading where fe- 
males nay be more likely to wear a uniform. Then again, sex may be re- 
lated to achievement , Females nay be better motivated to study than 
males at this age in vocationeQ. education programs and success as a pro- 
gram nay be related to enrollment of more hi^ly motivated female students. 

The number of students in a program is also related to success. It 
may be that the larger programs are more visible on campus because of 
their larger enrollments and staff, and presumably, larger budgets. 

The procedure of interviewing students prior to admission is also 
related to success. Again, this may be related to other items such as 
size and visibility. The selectivity suggested by an interview proce- 
dure for admission to a program is generally either a fxinction of a sur- 
plus of applicants over "seats” or an attenq>t to increase success in a 
program by admitting only those who are likely to ccanplete the coxirse 
of study. 

Student Visibility Ttemg 

These were mentioned above in connection with the number of students 
in the program. Successful programs appear to be programs where the stu- 
dents eu:e more visible because they wear distinctive clothing or because 
they tend to cluster together for course work and extracurricula activ- 
ities. This may be related to the success of the uniformed vocations 
such as nursing. It may be that these uniformed vocations stress exter- 
nal standards and prestige, as indicated by the high correlation of state 
licensing procedures (under Co\a*se Items in Table U), 

It seems clear that successful programs are those programs which have 



hi^er proportions of females, which encourage visibility through dis- 
tinctive clothing, which encovirage a solidarity among students, and 
which function with some degree of state approval. 

Course Items 

One of the three items we have grouped under this title is nega- 
tively related to success. Although the correlation of the provision 
for remedial work with success is not very strong, the sign suggests 
that program success is contingent upon admission of students who do not 
require remedial work prior to admission to the program. Perhaps the bet- 
ter students gravitate to the more successful programs; perhaps students 
who enroll in successful programs have already completed their remedial 
work before applying for admission to a specific occupational program. 

The use of off-campus facilities for class on a regularly scheduled 
basis as a correlate of success has to do with the glamour of the real 
world. Classes held in the field bring students into contact with prob- 
lems that eu:e relevant to the future occupation. 

Instructor Items 

The instructional staff cheuracteristics associated with success are: 
size of instructional staff and current experience in the program spe- 
cialty. Size of staff is related to the number of students in the major 
and the student-teacher ratio. A program with more Instructors offers 
the students a wider choice of Instructor vlet'fpolnt and personality and 
a wider view of Job alternatives. Recent experience in the field may be 
related to the use of off-campus facilities for regularly scheduled class 
work . 

Instructor Supervision Items 

The strength of association of items related to supervision of in- 
structors is not very strong. Nevertheless, these items point to the 
fact that successful programs seem to demonstrate a closer program man- 
agement-staff relationship. Classroom visitations are made on a regular 
basis and they are made impartially to the classrooms of both tenured 



and non-t enured teachers. 

Although vl ',0 visits the classroom does not make a difference, it 
is clear that teachers in progreuns identified as successful are subo 
Ject to more visits by a wider range of supervisory personnel than are 
teachers in programs not identified as successful. 

Advisory Committee Items 

The strength of association of the three advisory committee items* 
althou^ modest* points to the importance of the advisory committee to 
the identification of successful programs. It seems clear that success 
is contingent upon the number of committee members* the munber of com- 
mittee members who are cxirrently enployed in the occupation* and the 
meeting of the committee during the current school year. 

TESTS OF HYPOTHESIS I - Vocational Programs Identified by Delphi Panels 
as Most Successful Have Common Identifiable Characteristics 

T-tests were performed across "success” and "other" as groups for 
all variables that are continuous. The 17 tests can be identified in 
Table 3 by the presence of standard deviations for all continuous vari- 
ables. It will be seen that seven of these tests were significant at the 
.10 level or less. In all cases except one, the mean for the "success” 
group was higher than the mean for the "other'* group. The exceptional 
case was the item dealing with the male-to-female ratio in each program. 

Chi-square tests were performed on all dichotomous items. The re- 
sults are similar to those found with the t-tests. The thlrceen tests 
significant at the .10 level or less also had cozrelatlon coefficients 
which were significant beyond the .10 level. 

The evidence seems clear. Of all the items in the Phase II ques- 
tionnaire* the 20 items listed in Table have the greatest potential 
for describing successful programs. 

Before going on to a test of Hypothesis II* we will present re- 
sponses from other items which describe vocational educational programs 
in the colleges. 
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Program Itenagement Items 



Many items in the Phase II questionnaire did not lend themselves to 
statistical analysis of the kind used with continuous or dichotomous vari- 
ables. These items deserve mention because they describe some processes 
in vocational education which may merit attention in future research. 

The first group of tables have to do with simple counts of who in a 
management or supervisory capacity visits the classrooms of instructors. 

We determined above that items discussing visits to classrooms of teachers, 
both tenured and non-tenured, had the power to differentiate programs pri- 
marily because more visits were made to teacher's classrooms in successful, 
programs. Tables 5 and 6 Indicate who has the responsibility for making 
such visits for our sample. Although the items were significant, the fre- 
quency distribution of responsible persons revealed no basic differences. 
The person named most often as having responsibility for visits to class- 
rooms in both successful and other programs is the chairman of the depart- 
ment or specialty, with the vocational education dean named second most 
frequently. A scattering of other people were also mentioned. 



TABLE 5 

Frequency of Mention of Titles of Responsible Persons in Program Management: 
"By Whom are Tenured Teachers Visited?" Program Management Question 2-A-l. 



RESPONSIBLE PERSON 
Chairman and Instructor 
Voc-ed Dean 
Chairman 

Voc-ed Dean and Instructor 

Voc-ed Dean and Chairman 

Voc-ed Dean and Dean of Instruction 



SUCCESS OTHER TOTAL 

10 1 

7 5 12 

ll* 10 2U 

10 1 

10 1 

Oil 
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TABLE 6 



Frequency of Mention of Titles of Responsible Persons in Program Management : 

"By Whom are Non-Tenured Teachers Visited?" Program Management Question P-A-?. 



RESPONSIBLE PERSON 



SUCCESS 



OTHER 



TOTAL 



Chairman and Instructor 

Voc-ed Dean 

Chairman 

Voc-ed Dean and Instructor 

One or more of the Lay Committee 

Voc-ed Dean and Chairman 

Voc-ed Dean, Chairman, and Instructor 

Voc-ed Dean and Dean of Instruction 



10 1 

5 9 

16 11 27 

10 1 

10 1 

2 13 

Oil 
1 3 U 



A second group of tables have to do with responsibility for ten neces- 
sary management functions identified in question 1 of the Program Management 
section of the Phase II questionnaire. Statistical tests were not performed 
on these data, and they ore presented in Tables 7A through 7J for inspection 
and con^leteness . 



TABLE 7A 

Frequency of Mention of Titles of Responsible Persons in Program Management : 
"Who has Major Responsibility for Calling Staff Meetings of Instructors 
Teaching in the Program?" 




RESPONSIBLE PERSON 

Voc-ed Dean 

Instructor 

Chairman 

Voc-ed Dean and Instructor 
Voc-ed Dean and Chairman 
Other 
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SUCCESS 



OTHER TOTAL 



2 6 0 

1 3 h 

30 25 55 

112 
10 1 
112 



* 



f 



TABLE 7B 

Freq.uency of Mention of Titles of Responsible Persons in l^ogram Management: 
"Who has Responsibility for Developing a Budget for this Occupational Program?" 



RESPONSIBLE PERSON 


SUCCESS 


OTHER 


TOTAL 


Chairman and Instructor 


5 


2 


7 


Voc-ed Dean 


li 


5 


9 


Instructor 


5 


li« 


19 


Chairman 


18 


11 


29 


Voc-ed Dean and Instructor 


1 


3 


It 


Voc-ed Dean and Chairman 


2 


1 


3 


TABLE 7C 


Frequency of Mention of Titles of Responsible Persons 
"Who has Responsibility for Interviewing Prospective 


in Program Management: 
Staff?" 


RESPONSIBLE PERSON | SUCCESS | 


OTHER 


1 TOTAL 1 


Chairmem and Instructor 


1 


l4 


5 


Voc-ed Dean 


1 


8 


9 


Instructor 


1 


3 


l4 


Cliairman 


16 


15 


31 


Voc-ed Deeui and Instructor 


5 


2 


7 


Instructor and One or more cf the Lay Committee 


1 


0 


1 


Voc-ed Dean and Chairman 


8 


1 


9 


Voc-ed Dean, Chairmeui, and Instructor 


0 


1 


1 


Voc-ed Dean and Dean of Instruction 


2 


2 


It 



O 

.ERIC 
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TABLE 7D 



Frequency of Mention of Titles of Responsible Persons in Program Management: 
"Who has Major Responsibility for Evaluating Instructors?" 



RESPONSIBLE PERSON 


SUCCESS 


OTHER 


TOTAL 


Chairman emd Instructor 


0 


1 


1 


Voc-ed Dea^ 


7 


9 


16 


Instructor 


1 


1 


2 


Chairmein 


l6 


18 


3U 


Voc-ed Deem and Instructor 


1 


2 


3 


One or more of the Lay Committee 


1 


0 


1 


Instructor and One or more of the Lay Committee 


1 


0 


1 


Voc-ed Dean and Chairman 


6 


2 


8 


Voc-ed Dean, Chairman, and Instructor 


2 


1 


3 


Voc-ed Dean and Dean of Instruction 


1 


2 


3 



TABLE 7E 








Frequency of Mention of Titles of Responsible Persons in Progreun Management: 
' Who has Responsibility for Ordering Instructional Materials and Supplies?" 


RESPONSIBLE PERSON 




SUCCESS 


OTHER 


TOTAL 


Chairman and Instructor 




2 


0 


2 


Voc-ed Deem 




0 


3 


3 


Instructor 




8 


18 


26 


Chairman 




22 


lU 


36 


Voc-ed Dean emd Instructor 




0 


1 


1 


Instructor and One or more of the 


Lay Committee 


1 


0 


1 


Voc-ed Deem and Chairman 




2 


0 


2 



O 

ERIC 
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Frequency of Mention of Titles of Responsible Persons in Program Management: 
”Who has Major Responsibility for Initiating Curriculum Additions or Revisions 



RESPONSIBLE PERSON 


SUCCESS 


OTHER 1 


TOTAL 


Chairman and Instructor 


T 


U 


11 


Voc-ed Dean 


1 


1 


2 


Instructor 


7 


23 


30 


Chairman 


Ih 


5 


19 


Voc-ed Dean and Instructor 


2 


2 


L 


One or more of the Lay Committee 


0 


1 


1 


Instructor and One or more of the Lay Committee 


2 


0 


2 


Voc-ed Dean and Chairman 


3 


0 


3 



TABLE TG 








Frequency of Mention of Titles of Responsible Persons 
"Who has Major Responsibility for Developing a Class 


in Program Management : 
Schedule?" 


RESPONSIBLE PERSON | 


SUCCESS 


OTHER 


1 T0TA.L 


Chairman and Instructor 


5 


6 


11 


Voc-ed Dean 


1 


2 


3 


Instructor 


U 


11 


15 


Chairman 


21 


12 


33 


Voc-ed Dean and Instructor 


2 


3 


5 


Voc-ed Dean and Chairman 


2 


1 


3 


Voc-ed Dean, Chairman, and Instructor 


1 


0 


1 


Other 


0 


1 


1 



O 

ERIC 
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TABLE 7H 



Frequency of Mention of Titles of Responsible Persons in Program Management 
"Who has Responsibility for Making Teaching Assignments?" 



RESPONSIBLE PERSON 


SUCCESS 


1 OTHER 


TOTAL 


Chairman and Instructor 


5 


5 


10 


Voc-ed Dean 


• 2 




6 


Instructor 


2 


5 


7 


Chairman 


19 


18 


37 


Voc~ed Dean and Instructor 


3 


3 


6 


Instructor and One or more of the I»ay Committee 


1 


0 


1 


Voc-ed Dean and Chairman 


2 


0 


2 


Voc-ed Dean, Chairman, and Instructor 


1 


0 


1 



TABLE 71 

Frequency of Mention of Titles of Responsible Persons in Program Management; 


Who is Responsible for Calling Advisory Committee 
RESPONSIBLE PERSON 


Meetings?" 
SUCCESS 1 


OTHER 


TOTAL 


Chairman and Instructor 


1 


1 


2 


Voc-ed Dean 


16 


16 


32 


Instructor 


3 


3 


6 


Chairman 


11 


8 


19 


Voc-ed Deem and Instructor 


0 


3 


3 


One or more of the Lay Committee 


0 


2 


2 


Instructor and One or more of the Leiy Committee 


1 


0 


1 


Voc-ed Dean and Chairman 


3 


0 


3 


Voc-ed Dean, Chairman, and Instructor 


1 


0 


1 
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TABLE 7J 



Frequency of Mention of Titles of Responsible Persons in Program Managemetst: 
"Vfho is Responsible for Chairing Advisory Comnittee Meetings?" 



RESPONSIBLE PERSON 



SUCCESS i OTHER 



TOTAL 



Chairman and Instructor 0 
Voc-ed Dean 9 
Instructor 3 
Chairman 13 
Voc-ed Dean and Instructor 1 
One or more of the Lay Committee 7 
Instructor and One or- more of the Lay Committee 1 
Voc-ed Dean and Chairman 1 



1 1 

10 19 

2 5 

11 2h 

3 k 

6 13 

0 1 

0 1 



Although an in-depth analysis of the items described in the ten pre- 
ceding tables will not be attempted, a systematic pattei*n was observed 
which ought to be noted. The pattern can be summarized as follows: 



DIVISION CHAIRMAN EQUAL MENTION INSTRUCT^ 



A, B, C, E, F, C, H, 1, J 

(Mrlsipii chairmu is more 
often mentioned as Inttiatlng 
management activities in most 
mccesaful programs: 9 of K) 
functioos.) 


( In no management ftioction in 
most successful programs were 
instmetor and division ebainnan 
equally mentioned: 0 of K) 
functions*) 


D 

(Instructor is less often men- 
tioned as initiating manage- 
ment activities in most 
aiccessful programs: 1 of K* 
functions* ) 


(Division chairman is led 
often mcntloMd as tnirtating 
managemesit activities in 
programs not identified as 
most saccessfah 0 of K) 
ftmetioDS*) 


D» I 

(hstructor and division chab- 
man are less often mentioned 
as equally sharing management 
activities in programs not Id- 
entified as most successful: 

2 of 10 fnnetions*) 


A, B, C, E, F, G, H J 

(bstmetor is more often 
mentioned as initiatiag man- 
agement activities in prog- 
rams yt identified as most 
successful: 8 of K) functioos*) 
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Those instances are tallied where the instructor or the division chair- 
man has been mentioned more often as responsible for a management fvinction 
in Tables 7A-J. By "more often" here, we mean the relative frequency of men- 
tion for the saae category. In most cases, in both successful and other pro- 
grams, the chairman is mentioned more often than the instructor is mentioned 
for the sane function. However, the comparison we make here, refers to how 
often each of the categories, chairman or instructor, is mentioned in success 
programs compared with how often that same category is mentioned in other 
programs. 

There appears to be a difference in programs identified as successful 
and those not identified as successful in who initiates program management 
functions. In case of programs identified as successful, the division chair- 
man is more often the initiator of management activities than he is in pro- 
grams not identified as successful; whereas in programs not identified as 
successful, the instructor is more often identified. This is a striking dif- 
ference in view of the fact that these programs are drawn from the same col- 
lege canqpuses where one would assume that the management function would remain 
reasonably uniformed from program to program. 

Miscellaneous Items 

A third group of tables contain responses to three items which do not 
have the power to differentiate programs but are nevertheless of interest 
in themselves. 



TABLE 8 

Frequency of Response to Curriculum Question "Is an Entry CoTirse Re- 
quired of, or Recommended for, all Students?" 

RESPONSE CHOICE ( SUCCESS ( OTHER { TOTAL j 



Entry Course Required 

Entry Course Highly Recommended 

No Such Course 



21 18 

7 7 

8 11 



39 

lU 

19 
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TABLE 9 



Frequency of Response to Curriculum Question 5: "is a Final or Wrap-up 

Covirse Required of, or Recommended for, all Students?" 



RESPONSE CHOICE 



SUCCESS 



OTHER i TOTAL 



FlneJ. Course Required 



12 15 27 



Final Coxurse .Recommended 



^ 3 7 



No Such Course 



20 18 30 



Tables 8 and 9 are from the Curriculum and Class Scheduling sections of 
the Phase II questionnaire and describe the frequency of required entry or 
wrap-up courses for the various programs. Of the 72 programs we cein say that 
53 have a required or recommended entry course and 3^ have a required or rec- 
ommended final course. Some 19 programs have a "no such course" for entry, 
and twice as many, or 30 programs, have "no such course" for exit. There are 
no statlsticeuL differences between successful and "other" programs. 

Two other tables in this third group describe the timing of certain events . 



TABLE 10 

Frequency of Response to Advisory Committee Question 2: "When does the 

Advisory Committee Meet?" 



RESPONSE CHOICE 


1 SUCCESS j 


1 OTHER 


1 TOTAL 


Monthly 


1 


It 


5 


Semester or Queurterly 


lU 


9 


23 


Yearly 


10 


9 


19 


Upon Call 


10 


11 


21 


Other 


1 


3 


k 



TABLE 11 



Freq[uency of Response to: "When must this Occupational Program be Entered 

by the Student?" 



RESPONSE CHOICE 



SUCCESS 



OTHER I TOTmT 



Fall Only 
Spring Only 
Either Semester 
Other 



10 9 19 

2 0 2 

22 27 1*9 

2 0 2 



Table 10 describes the times when the program Advisory Committees meet, 
presumably on a routine basis. There are no apparent differences between the 
successful and "other" programs. Table 11 describes the prescribed time of 
entrance of students into the program. There are no reported differences. 

TESTS OF HYPOTHESIS II - Vocational Programs Identified by Delphi Panels eis 
Most Successful Differ from "Other" Programs in Program Characteristics 

Discriminant Analysis 

Previously, we discussed the strength of association of 58 items (vari- 
ables) with programs identified as successful. We then discussed tests of 
significance for the 58 items and concentrated on the 20 items vlilch individ- 
ually could statistically distinguish between the successful and "other" pro- 
grams at an acceptable level of significance (p <.10). While the methods of 
correlations, chi-square euid t-test, proved useful in identifying those vari- 
ables which had the most potential for separating programs, they also heui a 
drawback; they could only be used one at a time— they were univariate . In 
this section and in the section to follow, we will consider two techniques 
that can be used to "test" groups of items, techniques which are multivariate . 

In this section we will discuss the results of a technique called dis- 
criminant emalysls. Discriminant analysis is a method which identifies the 
basic ways in which groups differ from one another. In this procedure two or 
more items or veiriables are considered together. The items are weighted 
according to their ability to discriminate, and then the weighted coefficients 
are used collectively to separate groups (here, "success" and "other" programs) 
according to the laws of probability. After the discriminant weights or func- 



tions 60 % calculated for each progreun, each progreun can then be "scored" and 
assigned on the basis of probability to one group or another. In our case 
we con^iute the discriminant scores for groups of items for each of the 72 pro- 
greens and then use these scores to assign the program to either of two groups, 
"success" or "other". This assignment can then be tabled and compared with 
the identifications made through the Delphi technit^ue. A statistical test is 
made to indicate the "significance” of the difference between the weighted 
means of the groups of items. 

Clearly one cannot consider edl possible groups of the ^8 items available 
nor would it be desirable to do so. We have selected several groups of items 
(which include all the items when considered collectively) on 6in a priori 
basis. These are presented in Tables 12A-L. It should be remembered that if 
the items selected for our discrimineuit analysis technique had a perfect abil- 
ity to discriminate, we would identify all programs exactly as they were iden- 
tified and labeled by the Delphi panels. The classification would be perfect. 
No programs would be incorrectly classified by the discriminant amalysis, and 
6iny statistical test would be significant. Although perfect agreement is 
rarely, if ever, encountered with real data, the diagreun below shows the rela- 
tionship as it would appear in em ideal situation. 

In a case where the agreement was perfect, the identification matrix 
would be: 

IDEAL RELATIONSHIPS BETWEEN METHODS OF CLASSIFYING DATA 







DELPHI TECHNIQUE 












Percent 






Success 


Other 


Agreement 


DISCRIMINANT 


Success 


36 


0 


lOOJJ 


ANALYSIS 

TECHNIQUE 


Other 


0 


36 


100? 



We would have arrived at the same decision about each progratm by two 
different methods. If such were the caise and identification were perfect by 
the discriminant analysis technique, we would merely have to collect informa- 
tion by questlonnedre from each college on each program and emalyze it. There 
would be no future need for the more costly and time consuming Delphi process. 

Although perfect agreement of classification is desirable, it is not pos- 
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sl'ble with the data at hand. What is possible is to apply discriminant anal- 
ysis to groups of items and assess the technique's ability to agree with the 
a priori classifications derived by the Delphi process. Inspection of Tables 
12A-'L indicates that, for the twelve groups of items selected for analysis, 
the percentage of agreement runs from 50^^ in Table 12 J to in Table 12L. 

This may be Interpreted as a random classification in Table 12J where as many 
"other' programs were classified correctly as were classified incorrectly, 
and an almost perfect classification in Table 12L where ell but one on the 
"success" programs are correctiy classified. Pertinent data from Table 12A-L 
has been collapsed for comparison purposes and is presented in T&ble 13. 

A special rase is presented in Table 12A where the twenty items found to 
be significantly associated with "success", were subjected to discriminant 
analysis as a "group". This is a group in name only because the items do not 
represent a common factor or generic group. These 20 items taken collective- 
ly have the ability to classlf'/’ correctly 30 out of 36 "success" programs. 
Fxirthermore, 32 out of 36 "other" programs were classified correctly by both 
techniques. 

With discriminant analysis we have used a statistical technique which is 
related to regression analysis to classify programs on the basis of known char- 
acteristics into two groups. The two groups are then tabled and contrasted 
with the classification of programs by the Delphi technique as in Tables 12A-L 
and "mis-classifications" noted. We are not prepared to say which technique 
Is responsible for the mis-classlficatlcm of progreuns. We want to stress the 
large number of programs correctly cleisslfled by both techniques and offer this 
as evidence of the validity of the Delphi technique when applied to the iden- 
tification of successful vocational education programs in community colleges . 
Through the application of discriminant analysis to twenty easily obtained bits 
of data, we have been able to correctly classify five out of six programs in- 
dependently identified as successful and seven out of eight programs not so 
identified.' (See Table 12A). Hjis is a truly remarkable amount of agreement 
between two apparently dissimilar techniques applied to the diverse vocational 
education programs offered by community colleges. It appears that generaliza- 
tion with regard to program characteristics of successful programs is quite 
possible, and therefore, further study should prove fruitful. 



TABLE 12A 



Comparison of Program Classifications made by the Delphi Technique and by 
Linear Discriminant Analysis for Twenty Items Which are "Significant". 



DELPHI TECHNIQUE 



Percent 

Success Other Agreement 

DISCRIMINAirr 
ANALYSIS 
TECHNIQUE 



Success 


30 


6 


8356 


Other 


k 


32 


39? 



F(20,51) = 3.76 p<.01 



TABLE 12B 



Comparison of Program Classifications made by the Delphi Technique arid by 
Linear Discriminant Analysis for Five Items from the Program Identification 
Section. 



DELPHI TECHNIQUE 











Percent 






Success 


Other 


Agreement 


DISCRIMINAHT 


Success 


26 


10 


72% 


ANALYSIS 










TECHNIQUE 


Other 


8 


28 


78% 



F(5,66) = 5.5U p<.01 



TASLE 12C 



Comparison of Program Classifications made by the Delphi Technique and by 
Linear Discriminant Anedysls for Six Items from the Instructional Staff 
Section. 



DELPHI TECHNIQUE 



• 




Success 


Other 


Percent 

Agreement 


DISCRIMIKANT 


Success 


2U 


12 




ANALYSIS 

TECHNIQUE 


Other 


12 


2h 


61 % 



F(6,65) = 2.76 p<.05 



TABLE 12D 



Comparison of Program Classifications made by the Delphi Technique and by 
Linear Discriminant Analysis for Nine Items from the Equipment and Facili- 
ties Section. 



DELPHI TECHNIQUE 











Percent 






Success 


Other 


Agreement 


DISCRIMINANT 

ANALYSIS 


Success 


25 


11 


ON 

VO 


TECHNIQUE 


Other 


10 


26 


12% 



F(9,62) = 1.01 NS 





- 70 - 



TABLE 12E 



Comparison of Program Classifications made by the Delphi Technique and by 
Linear Discriminant Analysis for Three Items from the Program Management 
Section. 



DELPHI TECHNIQUE 



Success Other 



Percent 

Agreement 



DISCRIMINANT ^7 9 75^ 

ANALYSIS 

Other 9 27 75J« 



F(3,68) = 2.08 NS 



TABLE 12F 

Comparison of Program Classifications made by the Delphi Technique and by 
Linear Discriminant Analysis for Eight Items from the Curriculxim and Class 
Scheduling Section. 



DELPHI TECHNIQUE 











Percent 






Success 


Other 


Agreement 


DISCRIMINANT 


Success 


26 


10 


12 % 


ANALYSIS 

TECHNIQUE 


Other 


10 


26 


12 % 



F(8,63) = 1.97 NS 



O 

ERIC 
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TABLE 12G 



Comparison of Program Classifications made by the Delphi Techniq.ue and by 
Linear Discriminant Analysis for Twenty-One Items from the Recruitment 
and Placement Section. 



DELPHI TECHNIQUE 



DISCRIMINANT 

ANALYSIS 

TECHNIQUE 



Success Other 
Success 33 3 



Other 



29 



Percent 

Agreement 

92 % 

61% 



P(21,50) = 1.60 p<.10 



TABLE 12H 

Comparison of Program Classifications made by the Delphi Technique and by 
Linear Discriminant Analysis for Six Items from the Advisory Committee 
Section. 



DELPHI TECHNIQUE 



DISCRIMINANT 

ANALYSIS 


Success 


Success 

21 


Other 

15 


Percent 

Agreement 

56 % 


TECHNIQUE 


Other 


11 


25 


69 % 



F(6,65) = 1.72 NS 







- 72 - 



TABLE 121 



Comparison of Program Classifications made liy the Delphi Technique and by 
Linear Discriminant Analysis for Four Items Describing Recruiting Visits 
to High Schools. 



DELPHI TECHNIQUE 











Percent 






Success 


Other 


Agreement 


DISCRIMINANT 


Success 


26 


10 


72 % 


ANALYSIS 

TECHNIQUE 


Other 


12 


2U 


67 % 



F(U,67) = .97 NS 



TABLE 12J 

Comparison of Program Classifications made by the Delphi Technique and by 
Linear Discriminant Analysis for Six Items Describing Placement Services. 



DELPHI TECHNIQUE 











Percent 






Success 


Other 


Agreement 


DISCRIMINANT 


Success 


22 


ll» 


61 % 


ANALYSIS 










TECHNIQUE 


Other 


18 


18 


50 % 



F(6,65) = .2U ns 
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TABLE 12K 



Comparison of Program Classifications made by the Delphi Technique and by 
Linear Discriminant Ana.lysls for Eight Items Describing the Distinctive 
Visibility of Students. 



DELPHI TECHNIQUE 



Percent 

Success Other Agreement 



DISCRIMINANT 

ANALYSIS 

TECHNIQUE 



Success 30 

Other 7 



6 



29 



S3% 



Ql% 



F(0,63) = 3.72 p<.01 



TABLE 12L 

Comparison of Program Classifications made by the Delphi Technique and by 
Linear Discriminant Analysis for Three Items Delimiting Prerequisite, Inter- 
view, end Remedial Provisions. 



DELPHI TECHNIQUE 



Percent 







Success 


Other 


Agreement 


DISCRIMINANT 


Success 


35 


1 


97% 


ANALYSIS 

TECHNIQUE 


Other 


13 


23 


6k% 



F(3,68) * 3.86 p<.05 
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TABLE 13 



Suomaxy Table for Comparison of Delphi Technique with Linear Discriminant 
Analysis. Abstracted from Tables 12A-L. 



Item 


"Success" 


Percent 








Group 


or "Other" 


in Agreement 


F Ratio 


Of 


p Less Than 


A 


S 


83 


3.76 


20,51 


.01 




NS 


89 








B 


S 


72 


5.5U 


5,66 


.01 




NS 


78 








C 


S 


67 


2.76 


6,65 


.05 




NS 


67 






D 


S 


69 


1.01 


9,62 


NS 




NS 


72 








E 


S 


75 


2.08 


3,68 


NS 




NS 


75 








F 


S 


72 


1.97 


8,63 


NS 




NS 


72 








G 


S 


92 


1.60 


21 , 50 . 


.10 




NS 


81 








H 


S 


58 


1.72 


6,65 


NS 




NS 


69 






I 


8 


72 


.97 


»»,67 


NS 




KT 


67 








8 


6l 


CM 

• 


3,65 


NS 




NS 


50 








K 


S 


83 


3.72 


8,63 


.01 




NS 


81 








L 


S 


97 


3.86 


3,68 


.05 




NS 


61i 







Multiple Regression Analysis 

Thus far in the analyses, we have discussed the association (correla- 
tion) of questionnaire items with programs identified as "success" and 
"other" progreuDS. We have tested the means of individual items for each of 
the two kinds of identified programs , t-test and chi-square , and we have 
examined the ability of subsets of questionnaire items to classify programs 
independently of the Delphi technique (through linear discriminant analysis). 
In this section we will test the predictive ability of a subset of available 
items in a multiple regression model . Although we cannot meet all of the 
assumptions necessary for complete accuracy and confidence in the multiple 
regression model (e.g., the number of programs san 5 >led, N = 72, is not large 
enough for the number of variables we would like to "test"), we feel the re- 
sults of such a regression, however acceptable, are of sufficient interest 
to be reported. 




We will enter the 20 "significeuitly" correlated and tested items (see 
page ^ 3 ) into a m>altiple regression eqviation in a stepwise manner in an at- 
tempt to predict the dependent variable "success". The variables are enter- 
ed one at a time into the equation (via computer program BMD 02R) on the ba- 
sis of their contribution to the partial correlation of the (already entered) 
combined variables . Contributions to the partial correlation coefficient are 
tested at each step and variables are either "kept" in the equation or re- 
jected. 

The basic data for this regression are given in Table 1^. Acronyms have 
been provided for identification of items. The size of the coefficients (be- 
t£us) give an indication of the relative contribution made by each variable 
when all variables are considered together . It will be noted that the item 
"non-tenured visits" did not qualify for the equation. 

An analysis of variance based on the regression of the 19 entered vari- 
ables is given in Table 15. The F ratio of U.OU with 19 and 52 degrees of 
freedom, may be \ised to test the hypotheses that the 19 independent variables 
exercise no influence on the dependent veuriable, i.e., ® ^2 •***” ^19 “ 

The calculated F ratio is significant at p < .001. This of course, is not 
startling since ve have already determined that each variable examined inde- 
pendently had the ability to differentiate "success" from "other" progrsims. 
Taken collectively, ve would expect the 19 variables to reinforce each other. 

We would like to use this collective reinforcement to make the major 

point of this section. Multiple regression calculations produce a multiple 

correlation coefficient, R, which indicates the extent to which sill the 19 

items considered together account for variance in the dependent variable, here 

2 

success. Tlie square of R, or R , represents the proportion of variance in the 

dependent variable accounted for by the 19 independent items. These "multiple 

2 

Rs" and "multiple. R s" are given in Table I 6 where it is seen that six of the 
19 variables accoimt for about 50 percent of the variance in "success". The 
column labeled "Increase in R " shows the cumulative contribution of each item 
to the variance in this regression. We have arbitrarily drawn a line at the 
50 percent point to separate the contributors of large amoxints of variance in 
programs from the contributors of small amounts of variance.- It should be not- 
ed that whereeis the first six items account for 50.8 percent of the varleuice; 
the last 13 items accovint for less than 10 percent additional variance. Clear- 
ly, "success" or "other" programs can be best predicted by the first six items 
with diminishing and edmost negligible contributions made to prediction by the 
remaining 13. 




it 



Through the technique of stepwise multiple regression we have been 
able to account for about ^0 percent of the variance in successful programs 
with six items. This is a useful amount of varianct tq account for in an 
exploratory study and compares favorably with variance accounted for in be- 
havioral science studies in general. It tells us which of the large number 
of items we began with are the best predictors of success. Our chances of 
increasing our ability to predict, and thus to identify the key parameters 
of successful programs, will be enhanced if ve continue to study how success 
in vocationeil education programs is contingent upon the concepts embodied in 
these items. 

TABLE Ik 

STEPWISE REGRESSION TABLE FOR TWENTY ITEMS* 



Variables in the Equation (constant = -.179) 









Standard 


No. 


Acronym 


Coefficient 


Error 


1 


License 


.29k 


.117 


2 


Male Ratio 


-.002 


.002 


3 


Number of Instructors 


-.0l»0 


.036 


k 


Instructors with Experience in Last Two Years .079 


.031 


5 


Instructors with Recent Summer Experience 


.030 


.033 


6 


Off-Campus Classes 


-.269 


.123 


7 


Classroom Visits 


.043 


.031 


8 


Tenured Visits 


.028 


.029 


10 


Number of Students 


.002 


.001 


11 


Students Wear Uniforms 


.334 


.114 


12 


Students in Same Classes 


.212 


.187 


13 


Students Eat Lunch Together 


.061 


.122 


lU 


Students Work Together 


.236 


.189 


15 


Students Occupational Clubs 


.102 


.117 


16 


Pre-Program Intei^ew 


-.101 


.123 


17 


Remedial Course Work 


-.101 


.130 


18 


Advisory Committee Met Recently 


.129 


.l4l 


19 


Number of Advisory Committee Members 


-.007 


.011 


20 


Advisory Conaittee Members Qnployed 


.006 


.011 


Variables Not in the Equation 












Standard 


No. 


Acronym 


Coefficient 


Error 


9 


Non-Tenured Visits 


- — — 


- - - 



• Items found to differentiate "success" and "other" programs. 
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TABLE 15 

ANALYSIS OF VARIANCE TABLE FOR THE DATA OF TABLE lU 



Source of* Variation 


Of 


SS 


MS 


F Ratio 


Regression 


19 


10.727 


.565 


li.Oli* 


Residual 

TOTAL 

• p<.001 


71 


7.273 

18.000 


.ll»0 





TABLE 16 

SUMMARY TABLE FOR STEPWISE REGRESSION FOR TWENTY ITEMS 



Step 


Item 




Multiple 


Multiple 


Multiple 


No. 


No. 


Item Entered* 


R 


r2 


in r2 


1 


11 


Students Wear Uniforms 


.1*56 


.208 


.208 


2 


1 


License 


.556 


.309 


.102 


3 


10 


Number of Students 


.610 


.372 


.063 


k 


7 


Classroom Visits 


.657 


.1*31 


.059 


5 


ll» 


Students Work Together 


.693 


.1*80 


.01*9 


6 


k 


Instructors with Experience in Last 
Two Years 


.713 


.508 


.029 


7 


18 


Advisory Committee Met Recently 


.727 


.529 


.020 


8 


6 


Off-Campus Classes 


.731* 


.539 


.010 


9 


2 


Male Ratio 


.71*1 


.51*9 


.010 


10 


8 


Tenured Visits 


.71*7 


.558 


.010 


11 


12 


Students in Same Classes 


.753 


.566 


. .008 


12 


15 


Students Occupational Clubs 


.757 


.571* 


.008 


13 


3 


Number of Instructors 


.761 


.579 


.005 


ll» 


5 


Instructors with Recent Summer 
Experience 


.761* 


.581* 


.005 


15 


13 


Students Eat Lunch Together 


.766 


.587 


.003 


l6 


16 


Pre-Program Interview 


.768 


.589 


.002 


17 


17 


Remedial Course Work 


.770 


.593 


.001* 


18 


19 


Number of Advisory Committee Members 


.771 


.591* 


.001 


19 




Advisory Committee Members Employed 


.772 


.596 


.002 



* Item 9* "Non-Tenured Visits”, was not entered. See Table ll». 
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PHASE II - Summary 



The analyses presented in Phase II have been based upon a statistical 
examination of the data retumed on 72 Phase II questionnaires. The ques- 
tionnaires contained information on 36 vocational education programs identi- 
fied as successful by the Delphi technique and 36 programs selected from the 
pool of programs not so identified. 

Through slnq>le correlation, t-test, and chi-square techniques it was 
detemlned that 20 of the ^8 items suitable for such analyses correlated with, 
and differentiated successful programs from other programs at an acceptable 
level of significance. The 20 items were grouped under the six headings of 
Student, Student Visibility, Course, Instructor, Insti*uctor Supervision, and 
Advisory Committee, and their relationship to identified success in vocation- 
al education was discussed. Of particular Interest is the fact that the 20 
variables statistically associated with success can be divided among five ma- 
jor topical groups. Within the limits of the questionnaire, while not ex- 
haustive, these groups cover the major areas of school activity: Student, 

Curriculum, Instructor, Management, and "Trusteeship". 

Discriminant analysis and multiple regression were used to further ana- 
lyze the questionnaire data. Using discriminant analysis on various combina- 
tions of items we found that we could classify programs into two groups with 
a very high degree of success in replicating the classifications derived by 
the Delphi technique. Using the twenty "key" items, for instance, we were 
able to "correctly" classify 83 percent of the "successful" programs and 89 
percent of the "other" programs. Mxiltiple regression performed in a stepwise 
manner on the same twenty items showed that six items account for about fifty 
percent of the variance found in the 72 programs. These six variables should 
provide the nucleus for any future questionnaire designed to study the char- 
acteristics of successful vocational education programs. It is noted that 
these six variables came from five different sections of the questlonnedre . 

It sho\ild also be noted that if we chose the seven items explaining the most 
variance (about ^3 percent) we wo\ild have Included a sixth section of the ques- 
tionnaire. 

The conclusion to be drawn from these results is that the concept of suc- 
cess in conmunlty college vocational education programs as Identified by the 
Delphi technique is viable and multidimensional. Through the use of a well- 
designed questionnaire and appropriate statistical techniques, we have Identi- 



fied some of the dimensions. It has been found that success is dependent 
upon a variety of characteristics which span students, faculty, management, 
curriculum, credentioling, and advisory committees. 
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APPENDIX 1 



PLAN OF THE STUDY 



1971 

August 



September 



October 



November 



December 



1972 

January 



February 



Phase I 

1.0 Select project advisory committee 

1.1 Select project director and secretary 

1.2 Locate project director's office 

1.3 Arrange with Napa College for fiscal management 

2.0 Hold advisory committee meeting to brief comm.ittee members on project 

2.1 Select consultant on study design and statistical analysis 

2.2 Infora advisory committee of specific fiscal arrangements 
and provide for statistical report to the committee 

3.0 Hold Joint meeting of steering committee, project director, and con- 
sultant on study design 

3.1 Develop preliminary model of Phase I and Phase II Instruments 

U.O Conduct a planning conference including college research personnel, 
vocational education directors, and consultants on research design 

U.l Familiarize conference participants with project plan and 

objectives 

k.2 Through Delphi process, identify and categorize vocational edu- 
cation deans opinion of success characteristics 

5.0 Conduct educational audit of project 

5>1 Modify instruments and procedures in accord with suggestions 
of conference participants and educationeLL auditor 

6.0 Identify those who will serve on vocational education program evalua- 
tion panel in each community college 

6.1 Review study plans with presidents and deans of instruction 

6.2 Request cooperation of chosen panel participants through re- 
search officer or vocational education dean 

7.0 Develop consensus on three "most successful" vocational education 
programs on each college campus 

7.1 Categorize statements of success characteristics Identified 
by panel members 
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Fn£. /^e II 



March 



April 

May 



June 



July 



8.0 Develop Phase II plan for data collection 

8.1 Modify data collection instrument in accord with statements 
made by Delphi panels 

8.2 Randomly select three "othei’" vocational programs for compctrison 
9>0 Administer the Phase II staiidardized interview 

10.0 Carry out analysis of data 

10.1 Identify common characteristics of most successful programs 

10.2 Identify characteristics of most successful programs which 
differentiate from other programs 

10.3 Assemble findings in preliminary report 

11.0 Hold project evaluation conference 

11.1 Distribute copies of preliminary report to participants 

11.2 Evaluate significance of findings and solicit recommendations 
for change 

11.3 Schedule final educational audit 

12.0 Present final report to Chancellor's Office and Research Coordinating 
Unit 




5. V? 



APPENDIX 2 



ACTIVITIES AND CONFERENCE OBJECTIVES FOR RESEARCH PERSONNEL 



Thursday - p.g, 

1. Understand the purpose of the local panel and assist in identifying 
panel peirticipants. 



2. Review and agree upon the final form of the instrument for de- 
veloping panel consensus. 



3. Review and agree upon the time, procedures for administration * and 
data collection methods in the consensus formation process. 



4. Write a college plan for identifying "most successful" programs 
including the above characteristics. 



3. Understand the purpose and nature of the reliability check of the 
consensus formation instrument. 



6. Review the concept of operational definitions and understand the 
process to be used in data collection after "most successful" 
programs have been identified. 



Friday - a.m. 

1. Operationally define for processes of measurement * the criteria 
for successful programs developed by the vocational education 
deans. 



2. Acquaint vocational education deans with the format to be used 
in data collection. 



Friday - p.m. 



With the assistance of consultants , agree on a specific research 
design to be used in carrying out the project. 
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ACTIVITIES AND CONFERENCE OBJECTIVES FOR VOCATIONAL EDUCATION DEANS 



Thursday - p.n, 

1. Identify tnd list characteristics of "most sucr.essful" programs 
on 3 X 3 cards. 

2. Develop a project definition for vocational education programs. 



Friday - a.m. 

1. Rate the importance of characteristics identified on Thursday. 



2. Review and understand the instrument to be used in identifying 
"most successful" vocational education programs. 



3. Review and understand the methods of measurement to be used by 
the college researcher in identifying "most successful" programs 
and modify where necessary. 



ACTIVITIES AND CONFERENCE OBJECTIVES FOR CONFERENCE CONSULTANTS 

1. Assist the conference participants in carrying out the objectives 
of the conference. 



2. Develop a specific rese2urch design to be used in carrying out 
the project. 



3. Provide written report of conference proceedings 



f 



EXPECTED OUTCOMES OF THE FIELD STUDY OF 
VOCATIONAL EDUCATION PROGRAMS 



Minimum Expectations 

1. Confirm presently held beliefs about the component parts of a 
successful vocational education program. 



2 , Give visibility to three locally identified "most successful" 
programs in each i>articipating college. 



3* Identify occupational programs which are most frequently cited 
by community colleges as most successful. 



4. Identify occupational programs which are never cited as most 
successful. 



Maximum Expectations 

1. In addition to the above, dispel mistaken assumptions now held 
about relative importance of program charaoteristlcs. 



2. Identify and make known effective practices which could be 
experimentally introduced into other college programs. 



3* Make available to participating college vocational education 
deans a wealth of empirically derived management information. 





NORTHERN CALIFORNIA COMMUNITY COLLEGE RESEARCH GROUP 



PLANNING CONFERENCE AGENDA 



The following activities have been scheduled for the project plan- 
ning conference to begin "A Field Study to Determine the Characteristics 
of Most Successful Vocational Education Programs" in northern California 
community colleges. The conference will be held at Shasta College on 
October 28 and 29: 



Thursday, October 28 



1 *CO p.m. — 2:00 p.m. 



Meeting of the Steering Committee of the Northern 
California Community College Research Group 



2:00 p.m. - 2:30 p.m. Conference Orientation meeting in room 211 (attended 

by both vocational deans and research personnel) 

2:30 p.m. - 4:30 p.m. Identifying criteria thought to be associated with 

successful Vocational Education programs (attended 
by Vocational Education Deans only) 



2:30 p.m. - 4:30 p.m. Review and rehearsal of procedures to be used in de- 
veloping program selection panels at participating 
colleges (attended by college research personnel only) 

6:00 p.m. - 7:30 p.m. No host cocktails and dinner at the Holiday Inn. Dis- 
cussion of phase three of the Nor Cal attrition study 
(attended by all interested conference participants) 



Friday, October 29 

8:00 a.m. - 9:C0 a.m. Operationally defining criteria thought to be asso- 
ciated with successful Vocational Education Program 
(attended by college research personnel only) 

8:00 a.m. - 9:00 a.m. Assigning group ratings to criteria thought to be 

associated with successful Vocational Education Pro- 
grams (attended by vocational deans only) 

9:30 a.m. - 11:30 a.m. Developing common definitions and methods of measure- 
ment (attended by vocational education deans and col- 
lege research personnel) 

11:30 a.m. - 1:00 p.m. Lunch - Faculty-Student Dialogue Room in the Student 

Center (attended by all conference participants) 

1:00 p.m. - 2:30 p.m. Summary of Conference proceedings and discussion of 

specific data gathering technique (attended by col- 
lege research personnel only) 

The conference is scheduled to begin in room 211 of the library. Participants 

will be notified if there are any room changes. 
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PHASE I DELPHI IKSTliJinaiTS 



FIRST ROWiD 



NAME 



Identify up to five vocational education programs conducted by your 
community college which you cons'* *r to be the "most successful." 

The list of vocational education , .'^grams Is attached. 

1. Use your own judgment as to what constitutes "success." 

2. Please do not consult i^lth others. Your Independent judgment 
Is desired. 

3. For each program give a brief statement of reasons you think 
the program Is successful. Write as many as you can think of. 

List your selections for "most successful" vocational education programs 
below: 

1. Program Title: 

Reasons for Success: (Please list each reason separately. List 

as many as you believe are Important.) 



2. Program Title; 

Reasons for Success: (Please list each reason separately. List 

as many as you believe are Important.) 




O'} 



3. Program Title; 

Reasons for Success: (Please list each reason separately. List 

as many as you believe are Important.) 



4. Program Title; 

Reasons for Success: (Please list each reason separately. 

as many as you believe are Important 



5. Progr am Title; 

Reasons for Success: (Please list each reason separately. 

as many as you believe are important 



List 
.) 



List 

) 



Participant Number 



NAME 

SECOND ROUND 



The attached sheets identify vocational education programs nominated as 
"most succersful" and lists the reasons why panel participants made their 
choice. Please read over each of these and the reasons indicated for 
their success and then using your best judgment, select the three which 
you consider to be the "most successful." 



1. P rogram Title; 



2. Program Title; 



3. Program Title; 



Participant Number 



THIRD ROUND 



NAME 



On the following sheets the five vocational education programs identified 
as most successful by panel participants are listed along with reasons 
given by the panel partlcipaats for making their choice. Ifliat we would 
like you to do now is; 

A. Look over the reasons given by panel participants for 
selecting the program. 

B. Select the five reasons you would Judge to be most • 

Important to the success of the program. 

C. Rank the five statements in the blanks provided at the 
left of the page from 1 through 5 , with 1 the highest 
ranking (i.c., place a 1 by the most important success 
criteria) . 
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GENERALIZED RESULTS OF THIRD ROUND D?LP}!i 

1.0 SmXIlT CBABACTEQISnCS 
1.1 HECTS ABILI1Y 1£VEL 

Provides opportanl'ty for nonracedeodc stadents. 

A career program in tdiich an average student can succeed. 

Good for atttdents inclined towards applications rather than theory. 

Screening of students for the program. 

U2 MEETS INTERBSTS OF STllENTS 

10 or 15 tines the nuad>er accepted apply for the program •» vary popular» 

Offers quick opportunity for male and female stadents to become 
professional ( 3 semesters). 

Attracts students \dio might not be exposed to college. 

Mature vomen can gain skills. 

Many veterans like this program. 

Natural inter st of male stadents. 

Program is popular enough to draw qualified stadents. 

1.3 MEETS FBEPASATION LEVEL OF STOie^ 

Availability of in-service training for career advanceawnt. 

Helps emplojrees get promotions. 

In-servico training for local officers. 

This meets a demand for retraining, whore aethers, once their family 
is established, can supplement income and establish skills. 

The program takes a student with little skill or knowledge In the 
field and produces a s.tndent with skills and knowledge in a sop- 
hiaticated field. 

1*4 MEETS FDUNCIAL MEEDS OF STDIENTS IN SCHOOL 

Virtually all fire science students are employed as paid or Tolunteer. 
Has kept many frcn going on welfare. 

Fills need for individual accoupliafament in conparatively brief time. 
1.3 ENBOIdMEMT POTEETIAL 

Large growth of numbers of stadents in classes. 

Heavy enrolLnent. 



o 
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Heavy onrolliaent* 

1.6 NUMOEai 1 /HO GEM JOBS AND SATISFACTOlDf FliUlFOR/IANCE 

High rate of student success in the field. 

Near 100^ placing of grads. 

All graduates %dio wish to vork as technicians hove obtained jobs upon 
graduation} nany prior to graduation. 

Perfonaonce of graduates have established a reputation for excellence. 
Groat demand for graduates. 

1.7 SUCCESS OF STUDENTS (LOlf DROP OUT RATE) 

Selection process appears to be ezcollent and instruction fine since 
attrition is v*}ry low. 

High rate of success in program conqilotion. 

1.8 CQNTRIIJUTBS TO FEQSQNAL QUALITIES 

A program \dxich gives success to students \Aio have liad few 
successes in the past* 

Selevance of program to needs of students for eelf-estoem. 

Developaent of pride in student, of preparation and need for work 
lAich candidate will bo perfomiicg. 

2.0 JOB MAPZET CHAIIACIEDISTICS 

2.1 NUHOm OF ENTDOr LEVEL JOBS 

Should be openings in this field for a long time. 

Growing demand because of the ever growing population. 

In this field there seems to be more jobs than trained people. 

There is an increased need for qualified mechanics. 

2.2 ISSPANDING JOB AREA 

National interest and realization of need for stimulating and 
creative experiences for young children. Affords e^loyment 
opportunities in rewarding work with young children for persons 
not interested In business, etc. 

This is very iiiq;>ortant due to Increased leisure time (T,V, etc.) 

New profession. 
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Institations ore notably deficient in tbeir preparation of appealing r 
oppotizing and edible foodt Students vho cooplete this course Aall 
have learned the basics of all throe of the deficiencies and how to 
overcooe them. 



Due to ezpoj^dinc medical facilities in commini'tyy more murces as well 
as LVM's and medical assistants ore needed. 



2.3 ADVANCEMENT POTENTIAL 

Graduates have been promoted in positions of higher rosponsibility 
far ^ster than any of us on the staff ever anticipated at the 
initiation of the technician training program. 

Chances for advancement excellent. 

2.4 DELATED OPPODIUNITIES 

General training that is well-founded could efventaally lead to 
furthering the potential of the student after becoming employed to 
many directionsi medical , legal , constr. et«.. 

Provides opportunities for other professions yAxo need to become 
acquainted with (and use) computers. 

Often proves a stepping stone to another career. 

2.3 SALADIES AND VOEEJUd CONDITIONS 



Has semi-profoasiooal status. 

Working conditions on the job. 

High wage scale. 

Gives one professional status. 

is able to sot his own working hours. 
Excellent salaries. 

Good ;pay for young women. 

Good atmosphere to work in. 



2.6 






AL APFEilL 



Data ]^rooo8sing| advertised as a glamor industry » caused many people 
to reiquost career in data processing. 

Glamor of job as it relates to nnrsing. 
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2.7 COHUUNITY VALUES 

Promotoa stron(; coommity and area interest as a natural ciceting 
ground between residents and college. 

Values of coaounity favor law cnforcesient. 

Both students and local industry benefit from progrom. 

Program meets community need. 

3.0 CHAEACTERISTICf OF INSTIIUCTION 

3.1 QUALIFICATIONS 

Outstan ding staff ~ wide backgroimd. 

Dedicated faculty. 

Highly qualified staff. 

A very strong teaching staff. 

3.2 HEIATIQN TO VOCATION 

Cooperation with lai# enforcement agencies in developing cxirriculum. 
Go>>d industrial contact by instructors. 

Good instructor contact with community. 

3.3 HErmODGLOGY 

Systems tically trains students for, and to adjust to, very demanding 
occupation. 

Interesting class materia l/student projects his roles. 

Excellent intern program. 

Lack of instructor pressure seems to allow those students with 
motivation necessary for successful job placement to emerge and take 
full advantage of equipment as i/ell as knowledge of instructor. 

Practical experience in running a police station, patrolling an area 
with patrol cars and on foot plus classroom study. 

3.4 MonvAnaN of students 

Enthusiasm of students taking program. 

Espirit de corp by participants (togetbomess in progrtmi). 

High motivation of students enrolled in program. 
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Motivation is high because job proaotion is dependent . upon the 
acquisition of either a certificate or degree* 

3*5 ENTHUSIASM OP STAFF 

Instructors highly enthusiastic. 

Very cohesive unit. 

3.6 INTEREST OP STAFF DI SlVDOrrS 

Instructors take personal interest in their students. 

Instructors villing to help vdth counseling ond further education. 

The staff assists the student in finding employmont and does 
follov^up on graduates* 

Students ore readily placed since cocuninity needs for highly skilled 
secretaries renains greAt and the staff takes on active role in 
student placeacnt. 

Good studenb>faculty relations. 

Students believe they receive outstanding help and support for the 
instructional staff. 

Instructors care about students. 

Instructor is highly respected and relates to students. 

Instructor effort in placement. 

4.0 CnABACTEDISTICS OF PROGRAM CQNTQ^ 

4.1 RELEVANCE OP xTOXHlAM TO JOB ITJr.UIinEirrS 



Geared to vhat students and employees vant. 

Courses offered currently applicable to business and industry. 

Local business is involved in providing vork related experience 
stations for students* 

Practical vork experience. 

Orientation of student to job. 

4.2 FUSHBlLITr 

Does not limit the number of students in program. 

Day and errening offering to suit student needs. 



O 
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lixcellent option of classea ond concentrations* 

Certificate of achievement at various levels* 

Worker any enter labor oarkett leave for period of time and reenter 
vithout (Teat deal of difficuliy. 

Transferability to four year technical program if decide to continue* 
llaay short lenB possibilities for updating skills* 

Curriculum is broad enough to prepare students for a variety of jabs* 
Program is flexible ~ transfer or employaent* 

finplo}>Bent possibilities for various levels of training and skills* 

4*3 SIAMDAEDS 

Graduates are veil trained * do veil on national and state boaid exams. 
Offers license* 

A prescribed coarse of study vhich oust be net by students* 

Begolreaenta of anatomy and physiology and prerequisite of chemistry 
insure better etadents* 

4.4 FEtf COUnSBS LIEB IT 

Unique program (not videly available)* 

4*3 (331EDAL 

Excellent program* 

Program has grcvn markedly, 

Progrems veil organised* 

4*6 EZTQISIVE, CCUPUBTE 

Pull program, I.E*» Coorsos from beginning accounting through anditing* 

Qange of lav enforcesMuit subjeote and coorsos is oocqileto. 

Program d spe n ds h i g hl y on sosdemio aresusi for necessary support courses* 

The program canslsts of an intansa block typa of training program 
aimed at specific tests and job dsscriptionB. 
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5«0 FACILITIES AND EQUIFMCMT 

Excellent modezn ccmpater facilltiee. 

Excellent firing range facility. 



6.0 SUFPOBT 



Indicates college's interest and abili'fy to initiate nev progreaa 
Mid achieve inaaediate success. 

Promotes interest. 



Promotes college in the area as it appeals to the people as a 
"vorthy** investment of tax dollars. 

6.1 sQPPoni 0? co^sKmm, iNDUsmy and business 

Support from hospital and medical personnel. 

The various training directors for oil levels of govemaent service 
support the program. 

Favorable comments from local indnstxy •“ contributions of equipment} etc. 
The Board of Qealtors supports the program of classes. 

6.2 ADVisanr comminhy suppori 

Vory good advisory ceomittee. 

6.3 ATSrrNlSTHATIQNL US/UDERSHIP AND SUPPOBT 

Coordinator is excellent. 

Beceived strong adarintstaratige support. 

Good adninistration of the program. 

Siqioxlor faculty leadership through hood of deportment. 

Lesderahip 

6.4 FACDLTf SUPPOBT 

Established program that enjoys faculty support. 

6*3 FINANCIAL SUPPOBI 

Inexpensive program relative to cost of program offered at UCLA and 
Pacific colleges. 

Well liEv.«:x4d Wr allied Health Grants. 

Good budget siqpport for currency with "state of the art". 
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6«5 REFOlATiaK AND tiEDIA SOFFOItT 
Good repatatiott* 

Ciuqpos has oreaied an environooni that docs not dcvmgrado the progroa* 
Canpos polica publicize their own activity by' the role the^ 



blc 

6/17/72 
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PHASE II QUESTIONNAIRE 



Program Identification 

!• Name of the Progra m 

2. Number of units required in the major 

Number of yenre the program has been offered on camini s 

4. Does this program have a state licensiug or cextitication procedure? Ye s N o 

5. What is the ratio of males to females in the program? 

6. Is your best Job market within the college district or outside tlic college district? Within Outside 



Ipstmctional Staff 



!• In the o'v>iN^tional councs specified in the catalogue fo r (name 

of the how many different instructional staff memben are involved in teaching required coutscs? 

(Chech iIhS apv4opriate blank) 

1 instiuctor 4 iiastructors 



2 instnictors 5 instructors 



3 instructors More than 5 



iQuestion 1 is asked to determine 11 a relationship exists between the number | 
staff and the quality of the program J 

2« How many Instructois teaching in core courses for this occupational specialization have had full-time exper- 
ience in the field? 

Number of instructors with experience in the past 2 years ______ 

Number o! ir:stiuctors with experience in the past 4 years 

Number of instnictors with experience in the past 6 years 

3. How many instructors have had part»time experience or summer experience during the 1S71**72 school year? 

Number having pai^time experience during regular ______ 

school year 

Number having summer experience 
Equipment and Facilities 

!• In the opinion oi the staff member being Interviewedi what are the five major pieces of equipment necessary 
to train students for Jobs In this occupational specialization? Please list on the following page. 






i* 



(Essential equipment continued) 



2« How many of these five essential pieces of equipment are unavailable or in need of replacement in your 
training facilities? 



3 , 



(number) 

In the opinion of the staff member being interviewed, is the space proviJed forstudeoU in the training 
facilities adequate to provide essential training? 

Yes No 



If an inadequacy exists, please be specific as to tlie course, the facility, and the type of inadequacy. 



4. 



Is all the equipment required in the program provided by tiie college or must some of it be provided by 
the student? 

All provided by the college 



Some provided by the student 

If some equipment is provided by the student, describe the equipment and the cost to the student. 



There is some indication that progprams which require the student to commit him- 
self to the program to the extent of providing some of his own equipment may have 
the effect of only selecting students with a serious intent to pursue the major and, 
therefore, be more successful than programs without the requirement. 

5« Ace off-campus facilities regularly used in the training of students for this occupational specialization? 

Ye s No 

Please select the alternative which tells how off-campus facilities are used: 

Used in a regularly scheduled class 

Used irregularly for work experience students 

Used for field trips 
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Program Management 



U Who has major responsibility for this vocational program? On the statements bcl<;w of activities associated 
with program management, write the title in (vocational education dean, division chainnan, department 
bead, instructor) of the staff member who performs each of the activities* (Draw a line through any 
activity which b not performed* ) 

Management Activit y Responsible Person 

A. Calls staff meetings of instmctois teaching in 

the program 

B. Develops a budget for this occupational program 

C* Interviews prospective teaching staff 

D* Evaluates instnicton 

£• Ordeo Instructional materials and supplies 

F* Iniates curriculum additions or revisions 

G* Develops a class schedule 

H* Makes teaching assignments 

h Calls advisory committee meetings 

J* CbaiB advisory committee meetings 

IThis question is asked to determine the specific roles taken by staff! 
imembers in roost successful programs* j 

2. Are classroom visitations made regularly by supervisory peisoimel? Ye s N o 

A. If classrooms are visited, how often apd by whom? * 



How Often ? 



By Whom? 



Are tenured teachers visited? 

Are DOTHtenuxed teachers visited? 



Curriculum and Cl^ Scheduling 

1* Must this occupational program be entered by the student in either the fall or spring, or may he enter 
either semester? 

Fall only 
Spring only 

Either semester 
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f}^ 



* 



2« How many students declared this occupational program as a college major in 1970-71 school year? 

(number) 

3« Is a specific semester-by-semester couxse sequence made available in writing to tlic student (college 
catalogue or hand out sheet) upon entry to the program or are courses described only in terms of re- 
quirements? 

Course sequences available 

Course requirements oul>' are identified 

4« Is an entry course required of or recommended for all students? That is, is there one course which is 

required or recommended to the student in his first semester which orients him to the vocational program? 

Entry couxse required 

Entry course highly* recommended — 

No such course 

5« Is a final or wrap-up course required of or recommended for all students? This would be a course normally 
taken in the semester of completion of the occupational program which reviews what has gone before and 
prepares the student for Job entry* 

Final course required 

Final couxse recommended 

No such course 

6* Are the courses for this major scheduled as a block» where in a given semester the student must take all 
the prescribed couxse work| or scheduled individually with some degree of student selection? 

Courses arc sclicdulcd as a block 

Courses are scheduled at student selection __ 

7* Where did the last two curriculum changes for this occupational program originate and when did they 
take place? (Please show who was most instrrmcntal in the change* By change, we mean here the 
addrtlon of a new coursC| the modification of an existing couisei the droppitig of a course from the 
curricuhimi a change in the couxse sequence, or a change in the units and hours*) 

Change Originaled Change Made Date 

With one or more of the instnretron staff 

With the dcpaitment head or division 
chairman 

Widi the advisoxy committee 

With the comueling office 



Other source 



Rccniiting and Placement 






I 



Do membexs of the instnictional staff in this vocational major regularly visit high schools in the col lege 
district? 

Yes No 



2 . 



3 . 



4 . 



In your opinion, do members of the counseling staff specifically discuss this vocational major on regular 
visits to hlgn schools in the scivice area? 

Yes No 

Aie brochures distributed regularly to high sclioob and other student sources describing this occupational 
program? 

Yes No 



Is publicity rqgularly released through the nows media singling out and describing this vocational program? 



Ye s No 

5 * Does the student enrolled in this program have one or more means of being recognized as a member of 
this particular occupational training group? Check each of the following alternative means of recog- 
nition which typify the occupational student* 

a. Wears a uniform or work gear which Is identifiable 

b* Takes roost of his courses in the same general area 

c* Has most of his classes with the same students 

d* Eat lunch together with other students in the training program 

c. Take frequent field trips together with other students 

f* Has several classes where he works closely together with 
the same students in a lab or work experience situation 

g* Belongs to a club to which other students training for this 

occupation usually be long 

h* Other activities which identify the student with the program 
not listed* Please specify* 



6 * Does this occupational program have prerequisite course requirements which limit student enrollment? 
If 80| please describe briefly* 



O 

ERIC 
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i no 



7* If DO screening procedures are used, how are students treated who enter the program without appropriate 
skilb to profit from instruction? In other words, what specific remedial T)rcvisions are available to 
eutering students? 



8. Arc prospective students intcivicwed prior to admission to tliis occupational program? 

Yes No 



9. Are students in this occupational program given specific assistance in getting a job upon completion of 
the program by any oi the following services? 

A. By the college placement ott ice. Please describe the service rendered, 



B. General assistance from Uic instructional statf. Please describe the service rendered. 



C* Assistance from a spccitically designated staff member. Please describe the service reudered. 



D. Assistance from membcis of the advisory committee. Please describe the service rendered. 



£• O^er assistance not mentioned. 



10. Docs the possibility for spinout jobs exist with this vocational program? That is, docs the student qualify* 
for jobs through partial completion ol the training program? Please specify the job title and number of 
semesteis needed to qualify for the job. 

Job Title Semester Qualified 



Advisory Committee 

1* Does this program have on advisory oominit^ie which met during the 1971-72 academic school year? 

Yes No 
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2« Docs this advisoiy committee meet: 



Monthly 

Semester or quarterly 

Yearly 

Upon call 

3* How many members arc on tlic advisory committee? 



(number) 



4. Is the advisory committee responsible for more than this one occupational program? 

Yes No 



5. How many members o! the advisory committee are currently employed in or are supexvisois of personnel 
in this occupational specialization? 



(number) 

6. Did this advisory committee propose or review the lost two changes made in the educational ciimculuni 
of this occupational program prior to the change? 



7. How many times in the past year have advisory committee membets assisted in student placement? 



There is some indication that the strength of a program is closely related 
to the community support provided by an advisory committee. Tliesc 
questions are designed to determine the quality of support provided by 
the advisory committee* 



o 

ERIC 
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APPENDIX 6 



VOCATIONAL EDUCATION. DEA.NS DESCRIPTION OF 
PROGRAM CHARACTERISTICS ESSENTIAL TO SUCCESS 



1 . 



Student availability 



2 . 



A program with student appeal 



A student group or class that perceives the occupation 
as satisfying and reasonably exciting 



_Ho\r student feels about the program 



3 . 



U. 



Program attracts students 

Recruitment of appropriate students 

A degree of ''screening'* for entering students 

Permit students to start in programs at the level that 

presents a challenge to them 

Students screened for capability 

^udents placed in proper place in program 

M otivation of students 

A student motivated to learn by active involvement 



Students highly motivated by an outside force 
(Must complete course or program in order to get 
pay raise or promotion) 

Student organizations in the specific occupation 
for building group enthusiasm 



Have the student and prpgram become a part of student 
activities in such ways as clubs » dances, publications 



_5 • Student proficiency 
6. Successful Job placement 



7 . 



A Job at the end of the program 



Placement 



3 . 



9 . 



High Initial placement rate 

Good program development 
Specific measurable objectives 

A curriculxim that leads to competence with defined 

objectives 



10. The program itself - its relevance and need 

Job availability 

Heed 



An instructional program that provides meaningful 

and satisfying class activities 

C urriculum content reflects what goes on in industry 

11 . Student oriented curriculum 



12. A program that is not "bogged-dovm" with irrelevant pre- 
requisites that will take a semester or year to coaq>lete, 
or that are unnecessary but nice additions 

13 . Saleable skills 



Saleable skills for students who complete only a 

portion of program as one semester, two semesters, etc. 

Student employed in field trained for 

Students trained for Jobs with possibility of 

employment 

Student enployed in field other than one trained for 

PGM educates-trains student to obtain Job (which 

exists)... A Job student really wants to obtain 

PGM with useful content (will help student pursue 

and be successful with particular avocation) 

lU . Program flexibility 

^More than one entry place for students to enter program 

A program that will provide a saleable skill in stages. 

The student may become trained in a year or less for 
entry level employment and that miay improve on this 
skill after employment 

}^ide range of related opportunity, e.g., career ladder 

or spinoffs 

Transferability - Job ladder for when Job market is bad 

Educational experience included in program. Versatile 

curricul\im allowing for career adjustments 

A good program is one that has built-in flexibility- 

can be easily phased out, or phased in. Part-time 
people-outside resources 







15. Growth in program 

_l6. An instructioned area that resembles a Job station 

17 . A laboratory that demonstrates an atmosphere for learning 

18 . Staff involvement in recmitment, coxinseling, placement 

19. Staff involvement 

20. Individuals guidance help available to students in program 

21. Advanced placement procedures - articulation with high 
schools 

22. VJork experience offered in conjunction with program 
Work experience education 

_23. An active interested advisory committee 

ihi active advisory committee 

Advisory committee resource groups 

A dvisory committee determines content (realism) 

Advisory committee ''clout" in employing community 

_2U. Close relationship with the appropriate industrial 
community Involved 

A strong and meaaingfxil tie-in with the consximers of 

the saleable skill produced by the program: a. placement 
b. advisory committees 

_25. Positive news media information 

Goed PR program 

Feedback to class from students emi)loyed in the 

Industry 

_26. Industry and community support (involvement) 

C ommunity participation and support 
_27 • Administrative support 

Philosophy and goals of college consistent objectives 

of vocational programs (including economic com.) 

District philosophy 

Cooperative board and administration 

Support 

1 ' ■! 



2 B, Well qualified, enthusiastic staff (instructional) 

W ell qualified instructioneJ. staff with background 
and wide acquaintance with labor management per- 
sonnel in occupation 

Strong and dedicated faculty and coordinator who 

believes in what' they are doing 



Staff 

A n enthusiastic, competent, euid innovative instructor 

Technically qualified student oriented instructor 

T he instructor 

O utstanding instructional staff 

A teacher who auulates professionalism 

Instructional staff, professional qualifications, per- 
sonal qualifications 

Occupational experience of instructors 

29. Student/instructor ratio 

Teacher/pupil ratio maintained at a level consistent 

with good instruction 

30. Up-to-date equipment that reflects that used in industry 

F acility that has been developed by representatives 
of the industrial community 

Flexible facilities 

A dequate budget-facilities 

_ 31 > Good administrative management 

32. Advice 
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HORCAL VOLITIONAL EDUCATION STUDY 
First Audit Report 




1. IHIROWCnON 

The remarks that follow are baaed upon observations made at 
the Planning Conference held at Shasta College on October 28 and 
29 » 1971 and upon examination of written materials furnished to 
me since that time. It is obvious that more than this is needed 
to prepare a thorough analysis of the project to date. 

Nevertheless, it has been enough for me to offer some opin- 
ions and smoe suggestions. As I understand my assignment as 
Project Auditor, it is to stand aside and look at the project 
from some distance, in tiiS hope that observations from this per- 
spective might assist those intimately involved in the detailed 
activities to see a little more forest and a little less trees. 

It is in this spirit that I offer this report. I make no 
claim that it is unbiased. To the contrary, I think I am as 
anxious as any of you that the project succeed. 

Tour Chairman has suggested that this report should not be 
exhaustive. Therefore, on the basis of my observations and read- 
ing, 1 should like to submit for your consideration my impression 
of some of the strengths and weaknesses of the project to date, and 
some recooaaendations for activities in the coming months. I shall 
list ten strengths, ten wealsaesses, and ten recommendations (none 
in any priority order). 
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STRKWGTHS OF THE raOJBCr 

Let us consider first some of the strengths of the Project. 

1. Surely one of the greet strengths of the project is the 
NORCAL research group itself. The talent « enthusiasm and 
spirit of cooperation found in this group > already clearly 
demonstrated, assure that much of value will result, 

2. Another great strength of the project is the accomplish- 
xaent of getting Vocational Education deans and institu- 
tional researchers from over twenty colleges to spend two 
days together in mutual discussions. Benefits accruing 
from this interchange almost guarantee the success of the 
project, should all else fall. 

3. I think the Planning Conference itself should be listed as 
one of the strengths of the project. Mot only was the com- 
bined meeting of Vocational Education deans and researchers 
an acccoplishment, the planning and operation of the Con- 
ference were clearly first-rate. The objectives of clari- 
fylng purpose, instilling enthusiasm, and receiving inputs 
were net in good style. 

4. The positive approach incorporated into the project is 

^ certainly one of its prime assets. Too often the question 
'Vhat's wrong with It?" is asked, too seldom the question 
'Vhat's ri^t about it?" It Is not only refreshing that 
you are looking for outstanding vocational programs, it 
also will undoubtedly enhance cooperation. 

5. Another strength, related to the one Just mentioned is the 
visibility the project will give to vocational programs. 

1 am sure you find, as I do, that even in this enlightened 
age there exist a large segment of the population who are 
unaware of the community colleges' fine vocational programs. 
It la to be hoped (and expected) that reports emanating from 
this project will significantly reduce the size of that seg- 
ment of the population. 

6. Institutional research is often criticized for not ha*^ng 
enou^ direct effect on management decision-making. One 
of the strengths of this project Is its potential for pro- 
ducing a sizable impact in this direction. Plans for pro- 
viding college administrators with information of real 
pragsmtlc value to them ere certainly to be coomended. 

7. The project proposal was obviously written with considerable 
cate and forethought. Its clarity and structure are to be 
ccoamnded. In particular, the detailed time scheduling 
should prove of such help to all concerned. 




A 

S 
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STRENGTHS OF THE IROJBCT (continued) 

8* Let me nov couueut upon two items Prom the materials I have 
received since the conference. First » the eet of Guidelines 
for Phase 1 prepared by the Project Director show depth of 
thought end general clarity of organisation* They sliould 
prove of much help to the individual college personnel as 
they proceed vlth the identification of "most successful" 
programs. 

.9* As a second item from the written materials, the report from 
Shasta College on the reliability check for panel convergence 
is quite impressive. It seems to me sufficient to give the 
green light to prompt panel formation and Round One of the 
Delphi procedure. 

10. As a final coamient on strengths of the project, let me con- 
gratulate you on your choice of a project director. His ex- 
perience idLth management processes, particularly with Delphi 
procedures, his willingness to assume and cooq>lete detailed 
and arduous tasks, and his general spirit of cooperation all 
contribute greatly to the asset side. 

WBAKMB88IB OP THK PROJBCT 

The above list of strengths of the project is certainly not ex- 
haustive, but simply the ten that most stand out, to my mind, at 
this juncture. Every project has weaknesses* Let us now consider 
some of these. As you will note, some that I will list are closely 
related to some of the listed strengths, by necessity. 

1. The first weakness I should like to mention is an almost 
necessary outcoms of the firat strength listed above. While 
two dosen or more researchers working together can accomp- 
lish things s single person cannot, the reverse is also true. 

1 know of no way for a group of twenty or more people to carry 
out the details of a research project. It is conceivable that 
some MGRCAL msobers, excited alMMit the project, may with all 
good Intention place an impossible burden on the Director. 

It seems to me ln|K>rtant to underline the obvious fact that 
the project has one director, and that, once he has his tasks 
clearly formulated by this A^sory Caemilttee, any other di- 
rection of the project should bs vlearlotia* 
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HBAKKE5SES OP PROJECT (continued) 

2« Although mentioned above as b strength, the conference ?>eet> 

Ing of vocational education deans and researchers coul<^ ulti' 
mately become a veakness* It eeema clear that the deans arc 
highly supportive of the project and that at the conclusion 
of the project they will be furnished a wealth of Information. 
Between the tine of Identification of the 'hnost successful'' 
programs and the preparation of the final reports I see little 
eoimunlcation with the deans* Surely they can provide more 
input than they had the opportimlty td provide at the con* 
ference* I auapect their ego will require that they have the 
opportunity* 

3* Although the Project Director has many desirable attributes 
as described above, his lack of experience in coiammity col* 
lege research endeavors must be Hated as a veakness of the 
project* 

4* 1 am somewhat concerned about the overall scope of the project* 

The potential problems in data collection and analyeif*, ea* 
peclally for the phases of characteristics differentiation, are 
staggering* I think more thought needs to be given to delimits* 
tlon* 

5* X feel there Is some confusion (possibly Just In my own mind) 
between characteristics of a program and criteria by which a 
program Is Judged successful. I note in reading the Delphi 
results from the Shasta pilot study that more than one respon* 
dent indicated that they thou^t a program was successful be* 
cause entrance requirements "insured good students," while 
others indicated as a reason for success the fact that the 
program "takes students with little skills or knowledge and 
prodtices a student with skills and knowledge In a sophisticated 
field." Do we have a philosophical problem here? A semantics 
problem? A research problem? The project proposal states 
that the purpose of the project Is to "Identify and make avail* 
able for local college use criteria associated with most sue* 
oessful vocational education programs .,.*" Should the term be 
characteristics? 

6. As mentioned, I think the Phase 1 Guidelines are well done* 
However, Phase X os Indlcatted does not coincide with Phase I 
In the proposal. It Is true that the first phase In the pro* 
posal Is concerned with data analysis but some confusion could 
result. What is to be Phase II? or III? or ? 

7* Again referring to the Phase I Guidelines, while instructions 
generally seem carefully thought out and clear, I suspect that 
more instruction on collecting and analyelng the data for the 
Delphi rounds Is needed for those researchers who have had no 
experience with the technique. 
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UBAKMBSSBS OF THE FROJBCI (contiiroed) 

8« Although under* tondable at this point In the project, I think 
relatively too nuch tine and energy have been spent develop- 
ing guidelines for panel selection, Delphi procedures and identi- 
fication of '*iDost successful" programs. Not enough thought has 
been given to taethodology for Identifying and Astlngulshlng 
among characteristics of good programs In general and eharac- 
tcrlstlcs specific to particular programs. I felt this was a 
vaakness of the planning conference, and have not seen enough 
In materials received since that time to allay my fears In 
this regard. 

9. In this same vain, I tlilnk the proposed Instruments for elicit- 
ing opinions regarding presence of certain characteristics 
(furnished to me by the Project Director) , are Inadequate In 
thcaaelves as data collecting Instnuaents . While the proposed 
questionnaire Items rallect conaidorabla thought and many (If 
not moat or all) of them appear appropriate, my criticism at 
this point Is twofold: (a) analysis and further use of the 

proposed collected data are not spelled out, and (b) I would 
hope that you will not settle for oplnlonnalra type data ex- 
clusively. 

10. As e final point In dlacuaalng project weaknesses, I would 
mentlott the stringency of the time aebedule. It appears that 
you are now about a month behind schedule with what appears 
the moat arduous part of the project still ahead. It appears 
that the time schedule will have to be expanded, or the pro- 
ject delimited even more severely than suggested above. 

BBOOHMKHDmONS 



Let me now offer aooe specific suggestions for some activities in 
the near future. I understand you are to have a meeting of. the NORCAL 
reaearcfaera on January 21. That might be a very good time to consider 
some of the items I will mention. 

1. Make sure all college representatives know precisely how to 
collect, analyse and report the data for each rotmd of the 
Delphi process. 

2. Plan a one or two day masting including the Project Director, 
the NORCAL Chairman, the design and analysis consultants and 
possibly one or two other axperlaneed NCAGAL researchers to 
consider types of measures appropriate to Identifying and dis- 
tinguishing among program charaeterlstlea and the general pro- 
blem of collection and analysis of the resulting data. I 
would suggest that consideration be g^yen specifically to the 
practicality of the measure, as vail as to the standard virtues 
of reliability, validity and relavanca. 
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REOQMMBnJATIOllS (continued) 

3* In this e«ne regard (toeasureoenta for characterletics) , be 
sure that plana for data analyale -- choice of techniques, 
oooputer utllleatlon, fomata for reporting, etc. -- are 
thought throng carefully In advance, so that data collection 
vlll he appropriate to the later analysis. 

4. X note a vealth of Ideas regarding program characteristics 
springing front the Delphi procedures used In the Shaata run- 
throu^. Be sure that plana, for measurement of characteristics 
are flexible enough to Incorporate similar Ideas front other In'^ 
stltutlons. 

5. Give some thou^t to the difference between criteria associated 

vlth a progrsm and characterlstlce of a program. (To quote 
Webster — criterion: a standard for Judging; eharacterl8tlc|<: 

that which distinguishes) • Also glvs some thought to charac- 
terlstlcs (criteria?) applicable to all programs and those 
applicable only to certain specific programs. 

6. Plan carefully the selection of eon^arlson programs not Identl' 
flad as "successful." I have not seen In writing a specific 
outline as to how this will be done. The major reason for in- 
cluding It here Is to suggest that possibly this portion of 
the project can be delimited rather drastically without too 
much damage to the overall project. This of course asstsses 
the method Is designed carefully with generallzlbillty of 
findings kept In mind. 

7. The Vocational Education deans are supportive of the project. 
Take advantage of this and tap their knowledge, possibly In- 
cluding some Interviewing. 

8. Start giving thought to the format to be used In presenting 
information to administrators. While some sophistication Is 
probably In order In the methods stages of the study, findings 
must be presented to adsdnlstretore In a form they can read and 
understand, and In such a way that the Information iriLIl make 
sense to them. As the old proverb puts It, If one has to search 
for a naadle In a haystack. It Isn't likely he'll be able to 
make a stitch In time. 

9. I suggest you rethink the time schedule for the remalxider of 
the project. A careful review at this point should Indicate 
how much additional dallmltlng should ba done in order to 
reach the major objectives of the project on schedule. 

10. Finally, let me urge you to give all the support, both moral 
and physical, you can to your Project Director. Re will have 
a very busy Spring. 
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00HCLD8I0B 

This report has become more vordy than I originally Intended, 
and 1 apologize for this. I hope no one will infer from this that 
I have an overall negative feeling about the project. To the contrary, 
if 1 had to sunasarize in one vord my rating of the project to date, 
that vord vould have to be ’’great." 

Let me reiterate that the above remarks are observations from a 
distance. Where I am on the mark, please give appropriate considera- 
tion; vhere 1 am off, please ignore. 

Thank you for permitting me to audit your project. 



iciiitiiififilritiicitit 
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NORCAL VOCATIONAL EDOCAIION STUDY 
SECOND AUDIT REPORT 



IWRODDCnON 

Thla ift th« 8«coiid and last Audit Report of the 1971*72 NORCAL 
Vocational Bdvieatlon Project Study. The flrat report covered the 
period from Inception of the project through December 31, 1971 and 
vaa preaented to the Project Advlaory Coomlttee at their January lA 
meeting held at American River College. Thla aecond report covera 
the period from January 1, 1972 through the Evaluation Conference 
held at Shaata College on June 19 and 20. 

Since conalderable efforta were to have been made following 
the Evaluation Conference and reaulta of theae efforta were not 
available In time for the preparation of thla report, the Coomlttee 
la advlaed that tlie following ramarka ahould be conaldered with thla 
fact In mind. 

I ahould like to categorlae my remarks today Into five areaa: 

(A) Progreaa In Phaac I, CB) Progreaa In Phase XI, (C) the Bvaltia* 
tlou Conference, (D) some general comnenta, and (B) some reecaaenda- 
tlona for future activities. 

A. PROGRESS IN PHAflB t 

Althodgb running 2-3 months behind the time schedule as outllxMd 
In the project proposal. Phase I of the project appeared at the time 
of the Bvaloatlon Conference to be nearing completion In good style. 
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Follovlx^ sre sooe nore •peelflc obierv&tion* : 

(1) I was Inpretsed by the concern for relleblllcy of the 
Delphi methodology. The addition of two features (the 
dual panels and the probability fonaula) not required 
by the proposal gave additional weight to the already 
eetabllehed fact that the nroiect leaders will not set- 
tle for unsubstantiated conclusions In the study. 

(2) The notion of calculating the probability of two panels 
coBlng up with the number of cosnon identifications that 
they did Is creative and should be useful In convincing 
vocational education deans and others that the Delphi 
procedure la reliable. However, there Is some doubt as 
to the correctness of the foraula. I trust that this 
vlll be die^ied (as per letter of June 28 to the HORC&L 
chairman) . 

(3) The process for Identifying the comparison programe (not 
most successful) la not clear. According to the proposal, 
for eadi program Identified as "most successful," a ran- 
dom sas^le of oollegea which offer this program and did 
not have It selected as "most successful" will be used. 

The preliminary identifications Indicate some of the "not 
Identified" to be completely different from thoso In the 
'Wet eucceasful" list. If some method other than the 
random eealgnment mentioned above la to be employed, cer- 
tain controls and cautions are In order. 

(4) The "Tentative Ranking of Sueeaas Characteristics" sssswd 
an excellent approach to obtain Idaas to pursue In Phase 
II of the project. However, the "mean" figures are con- 
fusing and may tend to give the Impression that the data 
level la stronger than it In fact Is. 

B. Prograea In Phase II 

The progress in Phase II at the time of the Evaluation Con- 
feranca was obviously limited by the delays sncountared In Phase 
I. Hcsrevar, the aeem^llahBmnts In Phase II by that tins Indicate 
clarity of general direction and that the analyals portion of the 
project pertaining to Phase II is In good hands. A few more specific 
<Aservatlons: 
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Progreii In Fhaae XI (eontlmied) 

(1) Ths d\olce of an Interview with a "auccessful program" 
director seemed a wise choice. The Instrument used, al- 
thou^ soiaevfaat awkward from a keypunching standpoint, 
showed care and thoroughnM* In its design. 

(2) The analysis of the Instrument, using blserlsl and point 
blserlal correlations, was creative and competently per- 
formed. A note of caution In rastdlng too much precision 
Into this type of statistic is in order, however. Cor- 
relations baaed on dichotomies are clever and are useful, 
but have large standard errors. Conclusions reached from 
this type of analysis with no other substantiation should 
be Interpreted with caution --as hypotheses for further 
Investigation, rather than as established facta. 

(3) Althougjh, as mentioned, the direction for Phase II seems 
clearly established and the initial Inatruaent appears to 
have merit, there is need for data less subjective In 
character. Initial findings from Phase I suggested that 
more objective data about the allege, the faculty, the 
students , and the oonBunity mi i^t be useful In providing 
nanageme^ Information to deans. Althou^ I realise the 
time problems, 1 hope this feature of the project will not 
be buried and forgotten. 

(4) I have sone concern about differentiation between charac- 
teristics of successful vocational education programs in 
general and characteristics applicable only to certain 
apeeific programs or to certain categories of programs. 

For example, looking at some of those indicated in the 
preliaixuary anal]^is as poasibly having significance, 
"nuirf)er of inatoctors with full-time experience In the 
field" could conceivably be a general characteristic 
idille "ratio of males to famalea" la obviously mote Im- 
portant in certain programs than others. Poeeibly eooa 
type of factor analyaia would be helpful in this regard, 
but I should think that a subjective categorisation by a 
rasaarcher knowledgasbla of program contact and objectives, 
followed by analyses of sub-groups compared with the total 
would yield useful Information. 
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C, Th» Bvaluatlon ConfTence 

Th« Bvaluatlon Confartncet although not as veil attended aa 
the earlier Plaxmlng Conference » again denonetrated the dedica- 
tion of the college reaearchera and vocational edncatlon deana to 
produce a atudy of real value to the oollegea. Incooplateneas of 
data reduced a oaa w hat the effectlveneaa of the conference In evaluat- 
ing fi n d in g® of the project; neverthaleaa» the conference appeared 
to be very helpful in obtaining conaenaua aa Co dlrectlona for con- 
cluding the project aa approved and funded and for continuing ef- 
forta beyond that point on aoiae of the phaaea of the project that 
could not be completed in Che allocated time. 

A atimulating feature of the conference vaa Che Inclualon of 
reporta from repreaentativea of prograna deemed "moat aucceasful*" 
Theae preaantatlona indicated clearly that if findings of the pro- 
ject do not rank enthuaiaaa, competence and dedication of program 
peraonnel hl^ on the list of characterlatica of auccessful pro- > 
gram, the entire atudy will be suspect. 

D. Some General Obaervationa 

tat me now offer a fev coBnenta about Che project In general. 

(1) Hy first general oomment would have to be that, In spite 
of fruatrationa of time delays, some peraonnel problems, 
coaounlcation difficulties, and other assorted factors 
which plaguat any endeavor of this type, the project must. 

In ita overall context, be rated outstanding, another 
fine acocnpliahaMmt of the hl^ly regarded group of 
W&CAL researdiera. 
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8om Cmaral Observations (continued) 

(2) You may recall that In my esriior report to you I sug- 
gested that should all else fall» the project vould have 
to be rated a success because of the aceomplishment of 
getting deans and researchers to get together* The above 
premiae was certainly not valid* but observations at both 
Conferences reinforces ny earlier conviction that the Inter- 
change between the groups brought about by this project 
will have valt^e long after the project Is completed. 

(3) 1 would certainly be remiss If I did not In this report 
point out to you my admiration for the performance during 
the year of your chairman, Valter Brooks* Bis conceptualisa- 
tion of the project, his ability to sense at what point and 
In what direction to move, hla ability to cooninleate ideas, 
his talent synthesising and crystallising group thinking, 
his reasoned calm at times when panic seemed In order, and 
above all his dedication and spirit of eooperatlou are in 
large measure responsible for the success of the project. 

(4) Z should also like to Indicate my liqiresslon of the contri- 
bution of Dr. WllUma Morris* As a representative of the 
funding source, his attitude could have been one of moni- 
toring and control* To the contrary, his participation 
throughout the project was consistently one of helpfulness 
and support and assisted greatly In the creation of a 
climate In which those Involved could perform most ef- 
fectively. 

(5) Although 1 must confess to a feeling that some of Its 
work (sueh as revision of budget classifications) could 
have been better accomplished by a small sub-coomlttes, 

(but realising the probleom of connunlcatlon and trans- 
portation and therefors discounting this feeling) , 1 
think each member of the Project Advisory Committee Is 

to be highly commended for his (or her) generous contribu- 
tions of tlise, talent, and energy. 

(6) Among these accolades, lot ne Insert one note of caution 
about a possible outcome of the project which should stu- 
diously be avoided. It Is to be hoped that programs ehoasn 
os outstanding vocational education programs will receive 
positive publicity and much needed exposure. It le equal- 
ly to be hoped that the logical Isverse of this atatesient 
will not take place. Bvery caution should bs tdcen to in- 
sure that programs not selected by tbs processes used In 
Phase 1 of the project are not branded as Inferior. 
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B. HaeoiBaendattona 

Let ne close vlth aoiae sxiggestlons from this ventage point for 
future activities* I think nost of these reconaa n datlooa can be car- 
rled out, at least to soae degree, whether or not a continuation of 
the project Is refunded. 

(1) Continue to investigate sore objective neasures for Identi- 
fying characteriatiea of successful prograas. Be sure data 
is collected in a format amenable to appropriate analysis. 

(2) Consider some alternative analyses to the dlehotooous cor- 
relation technique (for example, divide data into two 
groups — successful and non-successful programs — , calcup 
late appropriate statistics and make tests for signifi- 
cant dlff ereoses) . 

(3) Study more thoroughly the Phase I Delphi rounds, in parti- 
cular reasons for major shifts of selections. 

(4) Prepare findings In form for use by college administra- 
tors, being careful not to denigrate non-selected programs. 

(5) Continue Investigations of programs which vera selected 

In some colleges, passed over In others. Report any find- 
ings from this e^eavor with considerable care. 

(6) Continue to strengthen the working relationships estab- 
lished between deans and researchers. 

(7) Pat yourselves on the back a little. You deserve it! 
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